Background light subtraction for infra-red image

Abstract

To provide a method of performing background light subtraction in an infra-red (IR) illuminated image depicting a scene.SOLUTION: A method comprises: providing a rolling shutter image sensor; providing an IR light source configured to be turned on and off; changing an on-off status of the IR light source a plurality of times while capturing an image; capturing two or more image frames each image frame including a first set of lines of pixels comprising image data captured with the IR light turned on, a second set of lines of pixels comprising image data captured with the IR light turned off; creating an IR-illuminated image; creating a non-IR-illuminated image; and subtracting background light from the IR-illuminated image using pixel values in the non-IR-illuminated image.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] This disclosure relates generally to background light subtraction in infrared (IR) images depicting a scene, and more specifically to background light subtraction in IR images using a non-IR image, both images being captured by a rolling shutter image sensor. [Background technology]

[0002] Infrared (IR) images can be advantageous in a variety of situations where visual information is not sufficient to detect or analyze objects or phenomena. Common examples include nighttime or low light conditions, where IR cameras can capture images in the dark, making them useful for surveillance applications, search and rescue, and other applications that require visibility in low light environments. An example includes using a camera with infrared lights to read license plates, road signs, or detect traffic cones in low light conditions. License plates, road signs, and traffic cones are typically made of reflective materials, such as retroreflectors, and return a lot of IR light. However, the problem is that strong light sources in the captured scene, such as vehicle headlights, can be directed toward the camera and cause scattering and reflections that cover the object to be detected, making it difficult for the analysis software to find and analyze the object (e.g., read license plates, determine the content of road signs, etc.).

[0003] One common solution is to use an optical bandpass filter that passes only IR wavelengths through the camera. The downsides of this solution can be that it is more difficult to align and focus the camera because there is less light in the overall image, the camera is completely dependent on the IR light to function because the camera cannot capture any other light, and the hardware components increase the manufacturing cost of the camera (the filter includes any mechanism for switching it on and off).

[0004] Another existing solution that overcomes the problems described above involves synchronizing the IR light on the camera with the frame capture of the camera's sensor. The IR light is switched on for one full frame and then switched off for the next full frame. The image without IR is subtracted from the image with IR. Operating the sensor at a high frame rate, e.g. 60 fps, allows for capturing video at half that frame rate, i.e. 30 fps, with background light subtracted from the IR image. However, the problem with this solution is that if the camera moves (e.g. a camera in a car) or the scene moves (e.g. the scene includes a road), the time difference between the capture of the two images only allows for slow movement (of objects and / or camera in the scene) for the background light subtraction to work well enough (depending on the zoom level and application specific requirements for e.g. level of detail, etc.).

[0005] Therefore, improvements in this context are needed. Summary of the Invention

[0006] In view of the above, it would be advantageous to overcome or at least reduce one or more of the above-described disadvantages as set forth in the accompanying independent patent claims.

[0007] According to a first aspect of the present invention, there is provided a method for performing background light subtraction in an infrared (IR) image depicting a scene, the method comprising: providing a rolling shutter image sensor having a plurality of pixel lines, the rolling shutter image sensor reading out pixel data from one pixel line at a time during image capture; providing an IR light source configured to be turned on and off, where when turned on, the IR light is configured to illuminate the scene; changing the on / off state of an IR light source multiple times while capturing an image using a rolling shutter image sensor; - capturing two or more image frames using a rolling shutter image sensor, the rolling shutter image sensor being configured to read out image data into two or more image frames that are interleaved in time, each image frame comprising: a first set of pixel lines containing image data captured with the IR light turned on; a second set of pixel lines containing image data captured with the IR light turned off; and and capturing two or more image frames, Includes.

[0008] The method further includes creating an IR-illuminated image based at least in part on a first set of lines of the two or more image frames, and creating a non-IR-illuminated image based at least in part on a second set of lines of the two or more image frames.

[0009] The method further includes subtracting background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image.

[0010] A rolling shutter image sensor is a type of image sensor that captures an image by progressively scanning a scene, row by row or column by column, rather than all at once. This means that the image is not captured all at once, but rather is composed of a series of individual scan lines. The advantage of a rolling shutter is that it allows for a faster image capture rate. Typically, rolling shutter image sensors are used to capture high dynamic range (HDR) images, where a rolling shutter image sensor can be used to capture a series of images in rapid succession with different exposures (e.g., some images with short exposure times and some images with long exposure times), and the images can then be combined to create an HDR image. However, a rolling shutter image sensor can be configured such that the first and second images are captured with the same or similar exposure times. This allows for the camera or objects in the scene to move at a significantly higher speed compared to when the IR light is switched on for one full frame and then switched off for the next full frame. Examples of rolling shutter image sensors include sensors manufactured by Sony and Omnivision, such as CMOS image sensors that support DOL WDR (Digital Overlap Wide Dynamic Range) functionality.

[0011] The term "temporally interleaved" in the context of this specification should be understood as the pixel lines for the second image frame being read out before the readout of the last pixel line into the first image frame. Thus, the rolling shutter readout is staggered (row interleaved) such that the readout of a pixel line in the second image frame may begin immediately or very shortly after the readout of the corresponding pixel line in the first image frame.

[0012] By changing the on / off state of the IR light multiple times while capturing images with a rolling shutter image sensor, each image of the at least two images includes pixel lines captured with the IR light turned on and pixel lines captured with the light turned off. Advantageously, the on / off cycle of the IR light can be synchronized with the exposure time and the time difference between the two and further captured images, which allows stitching an IR illuminated image that includes (mostly, or only) image data captured with the IR light turned on from a first set of pixel lines in each captured image. Similarly, it may be possible to stitch a non-IR illuminated image that includes (mostly, or only) image data captured with the IR light turned off from a second set of pixel lines in each captured image.

[0013] Thus, the difference between the capture time of the IR illuminated image and the capture time of the non-IR illuminated image can be small enough to allow subtraction of background light from the IR illuminated image using pixel values ​​in the non-IR illuminated image, even when movement of the camera and / or objects in the scene is substantial.

[0014] The term "background light" in the context of this specification should be understood as any light in an image that adversely affects the detection and analysis of objects of interest in the image. Background light can come from natural sources, such as the sun or moon, but typically background light comes from bright artificial light sources, such as vehicle headlights, spotlights, street lights, etc., which cause poor reflections and scattering in the image.

[0015] Typically, the non-IR illuminated image and the IR illuminated image may have the same resolution, which simplifies the subtraction since the pixel value of a pixel at coordinates (X,Y) in the non-IR illuminated image may be subtracted from the pixel value of a pixel at the same coordinates (X,Y) in the IR illuminated image, which may otherwise require a necessary scale conversion of the coordinate space in one of the images. If the IR illuminated image and the non-IR illuminated image are captured using the same color space, e.g., grayscale, RGB, CMYK, etc., the subtraction may be performed directly using element-wise operations. If different color spaces are used, such as the IR illuminated image being captured in grayscale while the non-IR illuminated image is captured in RGB, then a necessary conversion of the color space of one or both of the images may be required before performing the subtraction (e.g., converting the RGB image, and optionally the grayscale image, to the LAB color space before performing the subtraction).

[0016] In some examples, the subtracting step includes, for each pixel coordinate of at least a subset of the pixel coordinates in the non-IR illuminated image, subtracting a pixel value at the pixel coordinate in the non-IR illuminated image from a pixel value at a corresponding pixel coordinate in the IR illuminated image.

[0017] Advantageously, this may enhance detail, and the sense of fidelity, in the modified IR-illuminated image, since areas of the image that are not affected by, for example, strong headlights, do not need to be modified as described herein.

[0018] In some examples, the subset of pixel coordinates is selected based on the luminosity (e.g., brightness) of pixel values ​​in the non-IR illuminated image. Thus, in these examples, the method includes selecting the subset of pixel coordinates based on the luminosity of pixel values ​​in the non-IR illuminated image. For example, if the luminosity (e.g., brightness in LAB color space, or value in grayscale color space) of the pixel at (X,Y) exceeds a threshold luminosity (e.g., 80, 90, or any other suitable threshold in LAB color space, or 200, 215, or any other suitable threshold in grayscale color space), pixel coordinate (X,Y) is included in the subset, and not included otherwise.

[0019] In some embodiments, the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates in the non-IR illuminated image, multiplying a pixel value at the pixel coordinate in the non-IR illuminated image by a weight value and subtracting the weighted pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at the corresponding pixel coordinate in the IR illuminated image. In an example, subtracting a portion of the background light (10%, 50%, 60%, etc.) may result in less noise being added to the modified IR illuminated image compared to when the entire background light of the non-IR illuminated image is subtracted. Advantageously, the present embodiment allows for subtraction of different portions of the background intensity, such that the amount of background subtracted and the amount of noise added may be optimized for each scene.

[0020] In some embodiments, the step of capturing two or more image frames includes capturing a first image frame and a second image frame. In such embodiments, it is advantageous to synchronize the on-off cycle of the IR light with the time difference between the first image and the second image such that pixel lines in the first image captured with the IR light on correspond to as great an extent as possible to pixel lines in the second image captured without the IR light off, and vice versa. This is because the step of continuously changing the on-off state of the IR light is synchronized with the first time span t spanThis can be accomplished by including completing an on-off cycle of the IR light during a time span where the IR light is turned on for the first half of the time span and turned off for the second half of the time span. In that case, the readout of the pixel lines into the two images can be such that for the first image, the rolling shutter image sensor reads out the first pixel line at a first time instant t, and for the second image, the rolling shutter image sensor reads out the first pixel line at a first time instant t. This pattern can then be implemented to read out the first pixel line in TIFF2024109072000002.tif7170. This pattern can then be implemented to read out the first pixel line 0.5t after the corresponding pixel line has been read out for the first image. span Data may be retained for all lines of pixels in the first and second images so that the lines of pixels in the second image can be read out at a later point in time.

[0021] The exposure time for each pixel line may be less than 0.5 milliseconds. Even if it is a short exposure time, this still means that some pixel lines are exposed with the IR light both on and off. To avoid or limit these lines in the IR illuminated and non-IR illuminated images, advantageously some embodiments include capturing a third image frame. The readout of the pixel lines into three images may then be performed such that for the first image, the rolling shutter image sensor reads out the first pixel line at a first time instant t. For the second image, the rolling shutter image sensor reads out the first pixel line at a first time instant t. Read out the first pixel line in TIFF2024109072000003.tif7170. For the third image, the rolling shutter image sensor: Read out the first pixel line in TIFF2024109072000004.tif7170. t in 3 image settings span (i.e., the time span to complete an on-off cycle of the IR light) is t span Note that the time t can be the same as or different from t . This pattern then always occurs 0.33t after the corresponding pixel line has been read out for the first image.span A pixel line in the second image is read out at a later time and always 0.66t after the corresponding pixel line has been read out for the first image. span Data may be retained for all lines of pixels in the first, second and third images for reading out the lines of pixels in the third image at a later time, such that the line data from the first, second and third exposures may be interleaved in time.

[0022] In some embodiments that include capturing a first image and a second image, t span may be at least twice the exposure time of any pixel line in the two or more image frames, such that a pixel line is exposed and read out in a first image, and then a corresponding pixel line is exposed and read out in a second image. span Note that the exposure time is anywhere between two and four times the exposure time. In the two frame embodiment, the longer time span may result in fewer areas having pixel lines exposed with both the IR light on and off. The longer time span results in a larger time interval between frames, which may be sufficient when the movement of the camera or objects in the scene is not very high, or when the requirements for the level of detail in the image capturing the scene are not very high.

[0023] In some embodiments that include capturing a first image, a second image, and a third image, t span teeth, TIFF2024109072000005.tif7170. In this embodiment, an IR illuminated image may be created using only the pixel lines that are exposed with the IR light on. Additionally, a non-IR illuminated image may be created using only the pixel lines that are exposed with the IR light off.

[0024] In some embodiments, the method includes an initial step of determining a brightness value of natural light in the scene, and if it is determined that the brightness value exceeds a threshold brightness, the method is stopped, and if it is determined that the brightness value does not exceed the threshold brightness, the method continues. Thus, IR light is only used when the natural light present in the scene is insufficient to capture detail using only natural light as illumination.

[0025] In some examples, the rolling shutter image sensor is configured to have the same exposure time for every pixel line, which reduces the complexity and simplifies the process of creating IR-illuminated and non-IR-illuminated images.

[0026] In some examples, the rolling shutter image sensor is provided in a camera that captures a scene including a road. As mentioned above, the present disclosure allows background subtraction even when the scene changes rapidly. One example is a traffic scenario. In some examples, the method further includes detecting a license plate in the modified IR illumination image. Since the license plate may be a retroreflector and returns a lot of IR light, identifying the license plate may advantageously be performed using IR illumination. As a result, license plate detection may advantageously be performed with background subtraction according to the present disclosure.

[0027] According to a second aspect of the present invention, the above object is achieved by a system comprising a rolling shutter image sensor having a plurality of pixel lines, where the rolling shutter image sensor during image capture reads out pixel data from one pixel line at a time, an IR light configured to be turned on and off, where when turned on the IR light is configured to illuminate a scene, and an IR light control unit configured to change the on / off state of the IR light multiple times during image capture using the rolling shutter image sensor.

[0028] The rolling shutter image sensor can be configured to capture two or more image frames, where the rolling shutter image sensor is configured to read out image data into two or more image frames that are interleaved in time, each image frame including a first set of pixel lines that include image data captured with the IR light turned on and a second set of pixel lines that include image data captured with the IR light turned off.

[0029] The system may further include an image stitching unit configured to create an IR illuminated image based at least in part on a first set of lines of the two or more image frames, create a non-IR illuminated image based at least in part on a second set of lines of the two or more image frames, and subtract background light from the IR illuminated image using pixel values ​​in the non-IR illuminated image, thereby creating a modified IR illuminated image.

[0030] According to an example, the system further comprises an image analysis unit configured to detect a license plate in the modified IR illumination image.

[0031] According to a third aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium storing instructions for performing the method according to the first aspect when executed on a device having processing capability.

[0032] The second and third aspects may generally have the same features and advantages as the first aspect. It is further noted that the present disclosure relates to all possible combinations of features unless otherwise specified.

[0033] The above, as well as additional objects, features, and advantages of the present invention will be better understood from the following illustrative and non-limiting detailed description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings in which like reference numerals are used for similar elements and in which: [Brief description of the drawings]

[0034] [Figure 1] FIG. 13 illustrates an exemplary scenario in which a modified IR-illuminated image is created by subtracting background light from an IR-illuminated image using pixel values ​​in a non-IR-illuminated image. [Diagram 2] FIG. 13 illustrates a schematic diagram of the capture of two temporally interleaved images using a rolling shutter image sensor and modulated IR light. [Diagram 3] FIG. 13 illustrates a schematic diagram of the capture of three temporally interleaved images using a rolling shutter image sensor and modulated IR light. [Figure 4] FIG. 13 is a schematic diagram illustrating modulation of IR light and pixel lines in a first and second image that are exposed with IR light on, off, or both on and off, and how this affects a modified IR illuminated image. [Diagram 5] FIG. 10 is a schematic diagram illustrating modulation of IR light and pixel lines in the first, second and third images that are exposed with IR light on, off or both IR light on and off and how this affects the modified IR illuminated image. [Figure 6] FIG. 1 illustrates a schematic diagram of a system including a rolling shutter sensor and an IR light configured to be turned on and off. [Figure 7] FIG. 2 illustrates generally a method for performing background light subtraction in an IR-illuminated image depicting a scene. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the disclosure are shown. The systems, devices and modules disclosed herein are described in operation.

[0036] The present disclosure relates to the field of background light subtraction.

[0037] As explained above, in situations where visual information is insufficient, for example, at night or in other low light conditions, IR images can be advantageously used to detect and analyze objects in a scene. However, if the scene depicted in the IR image includes one or more strong light sources, such as street lights, car headlights, etc., these may cause scattering and reflections that cover the object to be detected, making it difficult for the analysis software to find and analyze the object in the IR image. The present disclosure aims to provide methods, systems, devices and software to be used for background light subtraction in such scenarios, particularly in scenarios where the object to be detected moves in the scene or when the camera capturing the scene moves.

[0038] FIG. 1 illustrates a schematic of a low illumination scenario including a vehicle having headlights and a license plate. In FIG. 1, an IR illuminated image 102 and a non-IR illuminated image 104 are created as described above and further below in connection with FIGS. 2-7. As seen in FIG. 1, the headlights in the IR illuminated image 102 reduce the visibility of the license plate, for example, due to scattering and / or reflections resulting from the headlights being directed towards the camera capturing the IR illuminated image 102. Advantageously, pixel values ​​from the non-IR illuminated image 104 may be used to subtract disturbing light from the IR illuminated image 102 so that the license plate may be easily detected and analyzed. In FIG. 1, a modified IR illuminated image 106 is shown in which background light from the IR illuminated image 102 has been subtracted using pixel values ​​in the non-IR illuminated image 104. In the modified IR illuminated image 106, the license plate is clearly detectable and may be analyzed to detect a registration identifier (AXIS1), which is an alphanumeric ID that uniquely identifies the vehicle or vehicle owner.

[0039] In some embodiments, pixel values ​​from the entire non-IR illuminated image 104 (e.g., all pixels in the non-IR illuminated image 104) are used for subtraction of background light in the IR illuminated image. In other embodiments, a subset of pixel coordinates is determined from the non-IR illuminated image such that when creating the modified IR illuminated image 106, background light subtraction is only applied to a corresponding subset of pixels in the IR illuminated image 102. The subset of pixel coordinates in the non-IR illuminated image 104 may be determined based on an analysis of the brightness of pixel values ​​in the non-IR illuminated image 104 such that a pixel's coordinates are included in the subset of pixel coordinates if the pixel's brightness exceeds a threshold. Limiting the set of pixels used for background subtraction may reduce noise added to the modified IR illuminated image 104 by the subtraction operation. The concept of selecting a subset of pixel coordinates for subtraction is illustrated in FIG. 1, where the coordinates of pixel data 108, 110 corresponding to headlights and glare from the headlights in the non-IR illuminated image 104 may be included in the subset. It should be noted that this subset of pixel coordinates 108, 110 is used as an example to facilitate explanation of the concept of selecting pixel coordinates based on intensity value.

[0040] In some examples, pixel values ​​may be weighted with a weighting value (e.g., between 0 and 1, such as 0.1, 0.5, 0.9, etc.) before being used for background light subtraction. Subtracting a portion of the background light (10%, 50%, 90%, etc.) from the non-IR illuminated image 104 may result in less noise being added to the modified IR illuminated image compared to if all pixel values ​​of the non-IR illuminated image were subtracted.

[0041] In some embodiments, the non-IR illuminated image 104 and the IR illuminated image 102 are captured using the same color space. In these cases, the subtraction may be performed directly using element-wise operations, for example, by subtracting the RGB values ​​of a pixel in the non-IR illuminated image 104 from the RBG values ​​of the corresponding pixel in the IR illuminated image 102. In other embodiments, one or both of the non-IR illuminated image 104 and the IR illuminated image 102 are converted to another color space, for example, the LAB color space, before performing the subtraction, such that the L channel of the non-IR illuminated image 104 is subtracted from the L channel of the IR illuminated image 102. The resulting corrected IR illuminated image 106 may or may not be converted back to the original color space of the IR illuminated image 102. If the non-IR illuminated image 104 and the IR illuminated image 102 are captured using different color spaces, the subtraction may be accomplished by converting one or both to a different color space before the subtraction. For example, if the IR illuminated image 104 is a grayscale image and the non-IR illuminated image 104 is an RGB image, the following steps may be performed:

[0042] 1. Convert the RGB image 104 to the LAB color space, which separates lightness (L) from the color (a and b) dimensions of the image. 2. Extract the L channel from the LAB image, which contains the lightness information. 3. Subtract the L channel values ​​from the pixel values ​​of the grayscale image 104 using an element-wise operation.

[0043] Any other suitable transformation and / or scaling of one or both of the non-IR illuminated image 104 and the IR illuminated image 102 may be performed before performing the subtraction.

[0044] In scenarios where the vehicle moves while being captured by the camera, or where the camera moves while capturing the images, the non-IR illuminated image 104 and the IR illuminated image 102 are advantageously captured close together in time, so that it is possible to use the non-IR illuminated image 104 and the IR illuminated image 102 to perform the subtraction. This can be achieved using a rolling shutter image sensor that supports capturing two or more images with a smaller time difference compared to the readout time of the full sensor (e.g., first capturing the non-IR illuminated image 104 and then capturing the IR illuminated image 102, or vice versa). The rolling shutter image sensor supports an interleaved configuration, which will now be described in relation to FIG. 2.

[0045] FIG. 2 illustrates diagrammatically the capture of two temporally interleaved image frames using a rolling shutter image sensor and a modulated IR light source.

[0046] The rolling shutter image sensor is configured to read out pixel data from one pixel line at a time during image capture. As a result, by modulating the IR light source such that it changes on-off state 206 multiple times during image capture with the rolling shutter image sensor, each of the image frames includes a first set of pixel lines 214 (dark in FIG. 2 ) that includes image data captured with the IR light turned on, and a second set of pixel lines 216 (white in FIG. 2 ) that includes image data captured with the IR light turned off. Advantageously, an IR illuminated image (see 102 in FIG. 1 ) can be created based at least in part on the first set of lines 214 of the two image frames 202, 204, and a non-IR illuminated image 104 can be created based at least in part on the second set of lines 216 of the two image frames 202, 204. Both images 202, 204 contain a pattern of lines of pixels captured with the IR light off (light stripes) and lines of pixels captured with the IR light on (dark stripes) due to the rolling shutter readout pattern in combination with the blinking IR light. The time difference 208 between the first image frame 202 and the second image frame 204 is defined as the time span t span By setting the pixel count 212 equal to (substantially) half of 210, or a multiple thereof, the patterns in the two image frames 202, 204 are out of phase. In other words, a first pixel line 212 in a first image frame 202 is read out at a first time instant t, and the corresponding first pixel line 212 in a second image frame 204 is read out at a second time instant t. 2, the IR light source is modulated with a square wave 206 (on / off), where one half of the period of the wave corresponds to the time difference 208 between the two frames. span 210 may be at least twice the exposure time of any pixel line in two or more image frames.

[0047] In an example scenario, the IR light cycles on and off atspan is 1-2 ms, the shutter time (exposure time) of the rolling shutter image sensor is 0.5 ms, and the row time (time difference between the first exposure of adjacent pixel rows / lines) is 0.015 ms. Other configurations are possible as well. This is advantageous because a short row time results in less distortion of moving objects in each image frame 202, 204 captured by the rolling shutter image sensor. However, the shutter time is not infinitely short, and the row time is typically shorter than the shutter time, so some of the pixel lines are exposed with the IR light both on and off. This is shown in FIG. 4. The pixel lines read out during time span 402 in both the first image frame 202 and the second image frame 204 are exposed with the IR light 206 both on and off. The number of pixel lines read out during time span 402 corresponds to the ratio of the shutter time to the row time. The pixel lines read out during time span 404 in both the first image frame 202 and the second image frame 204 are exposed to only IR ON or only IR OFF states.

[0048] As a result, as shown diagrammatically in Figure 4 as "Result" 406, an IR-illuminated image (or non-IR-illuminated image) created from the two captured image frames 202, 204 includes, for example, pixel lines that are exposed with the IR light both on and off during time span 402. This may still be sufficient to perform a subtraction to create a corrected IR-illuminated image, as described in connection with Figure 1. However, stripes of image data result in the corrected IR-illuminated image in a two-image setup as shown in Figures 2 and 4 that may, for example, degrade the quality of image analysis of the corrected IR-illuminated image.

[0049] This problem may be solved using a three image setup as now described in relation to Figures 3 and 5. The three image setup may allow the modulation frequency of the IR light to be matched with the time difference between the first, second and third image frames such that there is always one image frame out of three with 100% IR light on in every row of the image. Correspondingly, there is one image frame out of three with 100% IR light off in every row of the image.

[0050] 3 illustrates diagrammatically the capture of three temporally interleaved image frames 302, 304, 306 using a rolling shutter image sensor and a modulated IR light source. The rolling shutter image sensor thus supports a three-frame interleaved mode.

[0051] 2, the IR light source is modulated to change on-off states 306 multiple times during image capture with the rolling shutter image sensor, with each of the image frames including a first set of pixel lines 214 (dark in FIG. 3) that include image data captured with the IR light turned on, and a second set of pixel lines 216 (white in FIG. 3) that include image data captured with the IR light turned off. Advantageously, an IR illuminated image (see 102 in FIG. 1) may be created based at least in part on the first set of lines 214 of the three image frames 302, 304, 306, and a non-IR illuminated image (see 104 in FIG. 1) may be created based at least in part on the second set of lines 216 of the three image frames 302, 304, 306.

[0052] The time difference 308 between the first image 302 and the second image 304, and between the second image 304 and the third image 306, is taken as the time span t span_2By setting the pixel count to (substantially) equal to or a multiple of 1 / 3 of 310, any pixel lines that are exposed with the IR light both on and off may be avoided in the non-IR and IR illuminated images. Stated another way, in the embodiment of Figure 3, the rolling shutter image sensor is reading out the first pixel line 212 for the first image frame 302 at a first time instant t. For the second image frame 304, the rolling shutter image sensor is Reading out the first pixel line 212 in TIFF2024109072000007.tif7170, for the third image frame 306, the rolling shutter image sensor The first pixel line 212 is read out in TIFF2024109072000008.tif7170. This is then repeated for each pixel line in the three image frames 302, 304, 306. In some examples, span_2 may be at least six times the shutter time to avoid a line of pixels in each image frame being exposed with the IR light 306 both on and off.

[0053] The result is shown diagrammatically in Figure 5. Similar to that described in connection with Figure 4 above, some pixel lines in each image frame are exposed with the IR light 306 both on and off. However, by capturing the three image frames 302, 304, 306 with a time difference between the first image 302 and the second image 304, and between the second image 304 and the third image 306, equal to or a multiple of (substantially) 1 / 3 of the time span of the IR light 306 on-off cycle, these pixel lines may be avoided in the result 506. Thus, the result 506 represents an IR illuminated image (or a non-IR illuminated image) created from the three captured image frames 302, 304, 306, and may not include pixel lines that are exposed with the IR light both on and off.

[0054] 6 shows, by way of example, a schematic diagram of a system 602 adapted to perform background light subtraction in an IR-illuminated image depicting a scene. The functionality of the system is now explained in relation to FIG.

[0055] The system 602 includes a rolling shutter image sensor 606 having multiple pixel lines, and the rolling shutter image sensor during image capture reads out pixel data from one pixel line at a time. Thus, in S702, a rolling shutter image sensor 606 is provided.

[0056] The system 602 further comprises an IR light source 604 configured to be turned on and off, and when turned on, the IR light is configured to illuminate the scene. Thus, in S704, the IR light is provided. IR is electromagnetic radiation (EMR) having a wavelength longer than that of visible light. Thus, IR is not visible to the human eye. IR is generally understood to encompass wavelengths from about 1 millimeter (300 GHz) to the nominal red edge of the visible spectrum at about 700 nanometers (430 THz). The infrared light source may be used to enhance the ambient light available for conversion by the night vision device and to enhance visibility in the dark without actually using a visible light source. In some embodiments, the IR light source, and thus the method 700 shown in FIG. 7, is used only when the determined brightness value of the natural light in the scene does not exceed a threshold brightness. Any type of light sensor (e.g., a light meter, a light meter, or a photometer) may be used to determine the brightness value of the natural light in the scene. The threshold brightness may be configured based on the use case and the parameters of the rolling shutter image sensor. The threshold brightness may be set, for example, at 2 lux, 1 lux, 0.5 lux, etc.

[0057] Any suitable IR light source 604 may be used, such as a bulb, lamp or diode.

[0058] The system 602 further comprises an IR light control unit 608 configured to change the on / off state of the IR light multiple times during image capture using the rolling shutter image sensor at S706.

[0059] The rolling shutter image sensor 606 is configured to capture two or more image frames (as illustrated in FIGS. 2-3) at S708, and the rolling shutter image sensor is configured to read out image data into two or more image frames that are interleaved in time, such that each image frame includes a first set of pixel lines that includes image data captured with the IR light turned on and a second set of pixel lines that includes image data captured with the IR light turned off.

[0060] System 602 further includes an image stitching unit 610 configured to create (e.g., crop and stitch) an IR-illuminated image based at least in part on a first set of lines of the two or more image frames, S710, as illustrated in Figures 4-5, and to create a non-IR-illuminated image based at least in part on a second set of lines of the two or more image frames, S712. Image stitching unit 610 may be further configured to subtract background light from the IR-illuminated image, S714, using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (as shown in Figure 1 as reference 106).

[0061] In some embodiments, the system 602 further comprises an image analysis unit 612 configured to detect a license plate in the modified IR illuminated image. A rolling shutter image sensor may be provided in the camera 600 that captures a scene including a road.

[0062] In some embodiments, the system 602 further comprises a wireless transmitter 614. The wireless transmitter 614 may be used to transmit the modified IR illuminated image or data such as any detected license plates or other data from analyzing the modified IR illuminated image. The data may be transmitted for storage or further analysis at a server. The data may be received by a human operator trained to follow the data.

[0063] The system 602 may be implemented in a single device, such as the camera 600, in an example. In other examples, some or all of the different components (modules, units, etc.) 606, 608, 610, 612, 614 may be implemented in a server or cloud. In general, a device (camera, server, etc.) implementing the components 606, 608, 610, 612, 614 may comprise the components 606, 608, 610, 612, 614, and more specifically, circuits configured to implement their functionality. The described features in the system 602 may be advantageously implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, such as a camera, and at least one output device, such as a display. Processors suitable for executing a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and the sole processor or one or more processors or cores of any type of computer. The processor may be supplemented by, or incorporated in, an ASIC (application-specific integrated circuit).

[0064] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are envisioned. For example, the image analysis unit 612 may be configured to detect road signs. The output from the image analysis unit may be used to control the vehicle on which the camera 600 is mounted. It should be understood that any feature described in connection with any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Moreover, equivalents and modifications not described above may be employed without departing from the scope of the present invention, which is defined in the appended claims.

Claims

1. 1. A method for performing background light subtraction in an infrared (IR) illuminated image depicting a scene, comprising: Providing a rolling shutter image sensor having a plurality of pixel lines (S702), wherein the rolling shutter image sensor during image capture reads out pixel data from one pixel line at a time; Providing an IR light source (206, 306, 604) configured to be turned on and off (S704), where when turned on, the IR light source is configured to illuminate the scene; Varying the on / off state (206) of the IR light source multiple times while capturing images using the rolling shutter image sensor (S706); capturing (S708) three image frames (302, 304, 306) using the rolling shutter image sensor, the three image frames being captured in time sequence, including a first image frame (302) in the time sequence and a last image frame (306) in the time sequence, the rolling shutter image sensor being configured to read out image data into the three image frames interleaved in time such that a readout of a first pixel line for the last image frame occurs before a readout of a last pixel line into the first image frame, each image frame comprising: a first set of pixel lines that are exposed only when the IR light source is turned on; a second set of pixel lines that are exposed only with the IR light source turned off; a third set of pixel lines that are exposed with the IR light source both turned on and turned off; and capturing (S708) three image frames, including: creating (S710) an IR-illuminated image (102) using only the first set of pixel lines of the three image frames; generating a non-IR illuminated image (104) using only the second set of pixel lines of the three image frames (S712); subtracting (S714) background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (106); Including, changing the on / off state of the IR light source includes completing an on / off cycle of the IR light source during a second time span (t span2 ) (310); the IR light source is turned on for a first half of the second time span and turned off for a second half of the second time span; is greater than or equal to three times the exposure time of any image line in said three image frames; 3. The method of claim 2, further comprising: Capturing a first image frame (302), a second image frame (304), and a third image frame (306); reading out a first line of pixels (212) for the first image frame (302) at a first time instant t with the rolling shutter image sensor; The rolling shutter image sensor reading out a first line of pixels (212) for the second image frame (304); The rolling shutter image sensor reading out a first pixel line (212) for the third image frame (306); Including, A method wherein x and y are positive natural numbers.

2. 2. The method of claim 1, wherein the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates (108, 110) in the non-IR illuminated image, subtracting the pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at a corresponding pixel coordinate in the IR illuminated image.

3. The method of claim 2 , further comprising selecting the subset of pixel coordinates based on brightness of the pixel values ​​in the non-IR illuminated image.

4. 2. The method of claim 1, wherein the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates (108, 110) in the non-IR illuminated image, multiplying the pixel value at the pixel coordinate in the non-IR illuminated image by a weighting value and subtracting the weighted pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at a corresponding pixel coordinate in the IR illuminated image.

5. 2. The method of claim 1, comprising an initial step of determining a brightness value of natural light in the scene, and wherein if it is determined that the brightness value exceeds a threshold brightness, the method is aborted, and if it is determined that the brightness value does not exceed a threshold brightness, the method is continued.

6. The method of claim 1 , wherein the rolling shutter image sensor is configured to have the same exposure time for all pixel lines.

7. Detecting a license plate in the modified IR illuminated image. The method of claim 1 further comprising:

8. The method of claim 1 , wherein the rolling shutter image sensor is provided in a camera that captures a scene including a road.

9. The method of claim 1, wherein x=1 and y=1.

10. a rolling shutter image sensor (606) having a plurality of pixel lines, the rolling shutter image sensor during image capture reading out pixel data from one pixel line at a time; an IR light (604) configured to be turned on and off, where when turned on, the IR light is configured to illuminate the scene; an IR light control unit (608) configured to change the on / off state of the IR light multiple times while capturing an image using the rolling shutter image sensor (S706); A system comprising: The rolling shutter image sensor is configured to capture (S708) three image frames (302, 304, 306), the three image frames being captured in time sequence, including a first image frame in the time sequence and a last image frame in the time sequence, the rolling shutter image sensor is configured to read out image data into the three image frames interleaved in time such that a readout of a first pixel line for the last image frame occurs before a readout of a last pixel line for the first image frame, each image frame comprising: a first set of pixel lines that are exposed only with the IR light turned on; a second set of pixel lines that are exposed only with the IR lights turned off; a third set of pixel lines that are exposed with the IR light both turned on and turned off; and Including, The system further comprises: creating an IR-illuminated image using only the first set of lines of the three image frames (S710); creating a non-IR illuminated image using only the second set of lines of the three image frames (S712); Subtract background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (S714). The image stitching unit (610) is configured to: the IR light control unit is configured to complete an on-off cycle of the IR light during a second time span (t span2 ) (310); is greater than or equal to three times the exposure time of any image line in the three image frames, and the IR light is turned on during the first half of the second time span and turned off during the second half of the second time span; The rolling shutter image sensor comprises: capturing a first image frame (302), a second image frame (304), and a third image frame (306); reading out a first line of pixels (212) for the first image frame (302) at a first time instant t; reading out a first line of pixels (212) for the second image frame (304); reading out a first pixel line (212) for the third image frame (306); It is structured as follows: A system in which x and y are positive natural numbers.

11. 11. The system of claim 10, further comprising an image analysis unit (612) configured to detect a license plate in the modified IR-illuminated image.

12. A non-transitory computer readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 9 when executed on a device having processing capabilities.

Images