Background light reduction for infrared images

Abstract

This disclosure relates to background light reduction for infrared images, and more specifically to a method for performing background light reduction in an infrared (IR) illuminated image depicting a scene. The method includes: providing a rolling shutter image sensor; providing an IR light source configured to be turned on and off; changing the on-off state of the IR light source multiple times while capturing an image; capturing two or more image frames, each image frame including: a first set of pixel lines including image data captured when the IR light is on; a second set of pixel lines including image data captured when the IR light is off; creating an IR illuminated image; creating a non-IR illuminated image; and using pixel values ​​from the non-IR illuminated image to reduce background light from the IR illuminated image.

Description

Technical Field

[0001] This disclosure generally relates to background light reduction in infrared (IR) images depicting a scene, and more specifically to background light reduction in IR images using non-IR images, where both images are captured by a rolling shutter image sensor. Background Technology

[0002] Infrared (IR) imagery can be advantageous in various situations where visual information is insufficient to detect or analyze objects or phenomena. Common examples include nighttime or low-light conditions, where IR cameras can capture images in darkness, making them highly useful for surveillance applications, search and rescue, and other applications requiring visibility in low-light environments. One example includes using cameras with infrared light in low-light conditions to read license plates, road signs, or detect traffic cones. License plates, road signs, and traffic cones are typically made of reflective materials (such as retroreflectors) and return a large amount of IR light. However, the problem is that strong light sources in the captured scene (such as vehicle headlights) can shine directly into the camera, causing scattering and reflection that obscures the object being detected, making it difficult for analysis software to locate and analyze the object (e.g., reading license plates, determining the contents of road signs, etc.).

[0003] A common solution is to use an optical bandpass filter that only allows the camera's IR wavelength to pass through. The disadvantages of this solution may be that the camera is more difficult to align and focus because there is less light in the overall image; the camera is entirely dependent on the IR light that is in play because it cannot capture any other light; and the hardware components (the filter, including any mechanism for turning it on and off) increase the camera's manufacturing cost.

[0004] Another existing solution to overcome the problems discussed above involves synchronizing the IR light on the camera with the frame capture of the camera's sensor. The IR light is turned on for one full frame and then turned off for the next full frame. The image without IR is subtracted from the image with IR. The sensor is operated at a higher frame rate (e.g., 60fps) so that it can capture video at half the frame rate (30fps) while subtracting the background light from the IR image. However, the problem with this solution is that if the camera moves (e.g., a camera in a car) or if the scene moves (e.g., the scene includes a road), the time difference between capturing two images only allows (the scene and / or objects in the camera) to move at a low speed (depending on the zoom level and specific application requirements such as level of detail) to allow the background light subtraction to work sufficiently well.

[0005] Therefore, improvements are needed in this area. Summary of the Invention

[0006] In view of the foregoing, it would be beneficial to address or at least reduce one or more of the disadvantages discussed above, as set forth in the appended independent patent claims.

[0007] According to a first aspect of the present invention, a method is provided for performing background light reduction in an infrared (IR) image depicting a scene, the method comprising:

[0008] - Provides a rolling shutter image sensor with multiple pixel lines, wherein the rolling shutter image sensor reads out pixel data from one pixel line at a time during image capture;

[0009] - Provides an IR light source that can be configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate the scene;

[0010] - The IR light source is switched on and off multiple times while the image is captured using a rolling shutter image sensor;

[0011] - A rolling shutter image sensor is used to capture two or more image frames, wherein the rolling shutter image sensor is configured to read out image data for two or more image frames that are staggered in time, each image frame including:

[0012] ○ The first set of pixel lines includes image data captured when IR light is turned on;

[0013] ○ The second set of pixel lines includes image data captured when the IR light is turned off;

[0014] The method further includes: creating an IR illumination image based at least in part on a first set of lines from the two or more image frames, and creating a non-IR illumination image based at least in part on a second set of lines from the two or more image frames.

[0015] The method further includes: subtracting background light from the IR illumination image using pixel values ​​from the non-IR illumination image, thereby creating a modified IR illumination image.

[0016] A rolling shutter image sensor is an image sensor that captures an image by progressively scanning a scene line by line or column by column, rather than scanning the entire scene at once. This means that the image is not captured all at once, but rather constructed from a series of individual scan lines. The advantage of a rolling shutter is that it allows for a much faster rate of image capture. Typically, rolling shutter image sensors are used to capture high dynamic range (HDR) images, where the rolling shutter image sensor can be used to rapidly and sequentially capture a series of images with different exposures (e.g., one image with a short exposure time and one image with a long exposure time), and these images can then be combined to create an HDR image. However, a rolling shutter image sensor can be configured such that the first and second images are captured with the same or similar exposure times. This allows objects in the camera or scene to move at significantly higher speeds compared to when IR light is turned on for one full frame and then turned off for the next full frame. Examples of rolling shutter image sensors include sensors manufactured by Sony and Omnivision, such as CMOS image sensors that support DOL WDR (Digital Overlap Wide Dynamic Range).

[0017] In the context of this specification, the term "temporally staggered" should be understood as reading a pixel line for the second image frame before reading the last pixel line for the first image frame. Therefore, the rolling shutter readout is interleaved (row-staggered), such that the readout of pixel lines in the second image frame can begin immediately after reading the corresponding pixel line in the first image frame or shortly after reading the corresponding pixel line in the first image frame.

[0018] By repeatedly changing the on-off state of the IR light while capturing images using a rolling shutter image sensor, each of at least two images will include pixel lines captured when the IR light is on and pixel lines captured when the IR light is off. Advantageously, the on-off period of the IR light can be synchronized with the exposure time and the time difference between two or more captured images, allowing an IR-illuminated image (mainly or only) to be stitched together, comprising image data captured from the first set of pixel lines in the respective captured image when the IR light is on. Similarly, a non-IR-illuminated image (mainly or only) to be stitched together can be constructed, comprising image data captured from the second set of pixel lines in the respective captured image when the IR light is off.

[0019] Therefore, even if the camera and / or objects in the scene move significantly, the difference in capture time between the IR-illuminated image and the non-IR-illuminated image can be small enough to allow the background light to be subtracted from the IR-illuminated image using pixel values ​​from the non-IR-illuminated image.

[0020] In the context of this specification, the term "background light" should be understood as any light in an image that adversely affects the detection and analysis of objects of interest in the image. Background light can originate from natural light sources, such as the sun or moon, but typically it originates from bright artificial light sources that cause undesirable reflections and scattering in the image, such as vehicle headlights, spotlights, streetlights, etc.

[0021] Typically, non-IR and IR illumination images can have the same resolution. This simplifies subtraction because the pixel value of a pixel at coordinates (X, Y) in the non-IR illumination image can be subtracted from the pixel value of a pixel at the same coordinates (X, Y) in the IR illumination image; otherwise, a scaling transformation of the coordinate space in one of the images might be necessary. When capturing both IR and non-IR illumination images using the same color space (e.g., grayscale, RGB, CMYK, etc.), element-wise arithmetic can be used directly for subtraction. When using different color spaces (e.g., capturing IR illumination images in grayscale and non-IR illumination images in RGB), a necessary transformation of the color space of one or both images may be required before performing subtraction (e.g., transforming the RGB image (optionally, the grayscale image) to the LAB color space before performing subtraction).

[0022] In some embodiments, the reduction step includes: for each pixel coordinate in at least one subset of pixel coordinates in the non-IR illumination image, subtracting the pixel value at that pixel coordinate in the non-IR illumination image from the pixel value at the corresponding pixel coordinate in the IR illumination image.

[0023] Advantageously, this can increase the detail and fidelity in the modified IR illumination image because areas in the image that are not affected by, for example, strong headlights do not need to be modified as described in this article.

[0024] In some examples, a subset of pixel coordinates is selected based on the luminance (brightness, etc.) of pixel values ​​in a non-IR illuminated image. In these examples, the method therefore includes the step of selecting a subset of pixel coordinates based on the luminance of pixel values ​​in a non-IR illuminated image. For example, if the luminance of a pixel at (X, Y) (e.g., brightness in the LAB color space, or lightness in the grayscale color space) exceeds a threshold luminance (e.g., 80, 90, or any other suitable threshold in the LAB color space, or 200, 215, or any other suitable threshold in the grayscale color space), then the pixel coordinates (X, Y) are included in that subset; otherwise, they are not.

[0025] In some embodiments, the reduction step includes: for each pixel coordinate in at least one subset of pixel coordinates in the non-IR illumination image, multiplying the pixel value at that pixel coordinate in the non-IR illumination image by a weight value, and subtracting the weighted pixel value at that pixel coordinate in the non-IR illumination image from the pixel value at the corresponding pixel coordinate in the IR illumination image. In the example, reducing a portion of the background light (e.g., 10%, 50%, 60%, etc.) results in less noise being added to the modified IR illumination image compared to reducing all the background light in the non-IR illumination image. Advantageously, the presented embodiments allow for the reduction of different portions of the background intensity, making it possible to optimize the amount of background reduction and the amount of noise added for each scene.

[0026] In some embodiments, the step of capturing two or more image frames includes capturing a first image frame and a second image frame. In such embodiments, it is advantageous to synchronize the on-off period of the IR light with the time difference between the first and second images, such that pixel lines in the first image captured when the IR light is on correspond as much as possible to pixel lines in the second image captured when the IR light is off, and vice versa. This can be achieved by a step of continuously changing the on-off state of the IR light, which includes a first time span t... span The IR light is turned on and off during this period, wherein the IR light is turned on during the first half of the time span and turned off during the second half of the time span. Then, the pixel lines of the two images can be read out, such that for the first image, the rolling shutter image sensor reads out the first pixel line at the first time point t, and for the second image, the rolling shutter image sensor reads out the first pixel line at the first time point t. Read the first pixel line. Then, this mode can be maintained for all pixel lines in the first and second images, such that the pixel lines in the second image are read 0.5 t after the corresponding pixel lines in the first image are read. span The time point was read out.

[0027] The exposure time for each pixel line can be less than 0.5 milliseconds. Even though it is a very short exposure time, this still means that some pixel lines will be exposed while the IR light is on and off. To avoid or limit these lines in both IR-illuminated and non-IR-illuminated images, it is advantageous that some embodiments include capturing a third image frame. The pixel lines of the three images can then be read out, such that for the first image, the rolling shutter image sensor reads out the first pixel line at a first time point t. For the second image, the rolling shutter image sensor... Reading out the first pixel line, and for the third image, the rolling shutter image sensor in Read the first pixel line. Note the t setting in the three-image settings.span (That is, the time span used to complete the on-off cycle of the IR light) can be compared with t in the two image settings. span They can be the same, or they can be different. Then, this pattern can be maintained for all pixel lines in the first, second, and third images, such that pixel lines in the second image are always 0.33 t after the corresponding pixel line is read from the first image. span The time point is read out, and the pixel line in the third image is always 0.66 t after the corresponding pixel line in the first image is read out. span The timing points are read out. Therefore, the line data from the first, second, and third exposures can be staggered in time.

[0028] In some embodiments, when capturing a first image and a second image, t span It can be at least twice the exposure time of any pixel line in two or more image frames. Therefore, a pixel line in the first image is exposed and read out before its corresponding pixel line in the second image is exposed and read out. It should be noted that in some embodiments, t span This is any multiple between 2 and 4 times the exposure time. In a two-frame implementation, a longer time span can result in fewer areas having pixel lines exposed as the IR light is turned on and off. A longer time span will result in larger time intervals between frames, which may be sufficient when the movement of the camera or objects in the scene is not so high, or when the requirement for a high level of detail in the captured image of the scene is not so high.

[0029] In some embodiments, when capturing a first image, a second image, and a third image, t span It can be set to make The exposure time is ≥ 3 times that of any pixel line in three image frames. In this embodiment, an IR-illuminated image can be created using only the pixel lines exposed when the IR light is on. Furthermore, a non-IR-illuminated image can be created using only the pixel lines exposed when the IR light is off.

[0030] In some embodiments, the method includes an initial step of determining a luminance value of natural light in the scene, wherein the method is terminated when it is determined that the luminance value exceeds a threshold luminance, and the method continues when it is determined that the luminance value does not exceed the threshold luminance. Accordingly, IR light is used only when the natural light present in the scene is insufficient to capture details using only natural light as illumination.

[0031] In some examples, the rolling shutter image sensor is configured to have the same exposure time for all pixel lines. This reduces complexity and simplifies the process of creating IR-illuminated and non-IR-illuminated images.

[0032] In some examples, a rolling shutter image sensor is provided in the camera capturing a scene including a road. As mentioned above, this disclosure enables background subtraction even when the scene changes rapidly. One example is a traffic scenario. In some examples, the method further includes the step of detecting a license plate in a modified IR-illuminated image. Advantageously, license plate identification can be accomplished using IR illumination, since license plates may be retroreflectors and return a large amount of IR light. Therefore, according to this disclosure, background subtraction can be advantageously used to achieve license plate detection.

[0033] According to a second aspect of the invention, the above-mentioned objective is achieved by a system comprising: a rolling shutter image sensor having a plurality of pixel lines, wherein the rolling shutter image sensor reads pixel data from one pixel line at a time during image capture; an IR light configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate a scene; and an IR light control unit configured to change the on-off state of the IR light multiple times while the image is captured by the rolling shutter image sensor.

[0034] A rolling shutter image sensor can be configured to capture two or more image frames, wherein the rolling shutter image sensor is configured to read out image data for two or more image frames that are staggered in time, each image frame including: a first set of pixel lines, including image data captured when the IR light is on; and a second set of pixel lines, including image data captured when the IR light is off.

[0035] The system may further include: an image stitching unit configured to: create an IR illumination image based at least in part on a first set of lines from two or more image frames; create a non-IR illumination image based at least in part on a second set of lines from two or more image frames; and subtract background light from the IR illumination image using pixel values ​​from the non-IR illumination image to create a modified IR illumination image.

[0036] According to the example, the system further includes an image analysis unit configured to detect license plates in a modified IR illumination image.

[0037] According to a third aspect of the invention, the above-mentioned objective is achieved by a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed on a device with processing capabilities, implements the method according to the first aspect.

[0038] The second and third aspects can generally have the same features and advantages as the first aspect. It should be further noted that, unless otherwise expressly stated, this disclosure relates to all possible combinations of the features. Attached Figure Description

[0039] The above and additional objects, features, and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of embodiments of the present disclosure with reference to the accompanying drawings, in which the same reference numerals will be used for similar elements, in which:

[0040] Figure 1 An example scenario is shown where a modified IR illumination image is created by subtracting background light from an IR illumination image using pixel values ​​from a non-IR illumination image.

[0041] Figure 2 The illustration schematically demonstrates the use of a rolling shutter image sensor and modulated IR light to capture two images that are staggered in time.

[0042] Figure 3 The illustration schematically shows the use of a rolling shutter image sensor and modulated IR light to capture three images that are staggered in time.

[0043] Figure 4 The modulation of IR light, pixel lines in a first and second image exposed with IR light on and off, or with IR light on and off, are schematically illustrated, and how this affects the modified IR-illuminated image.

[0044] Figure 5 The modulation of IR light, the pixel lines in the first, second, and third images exposed with or without IR light, and how this affects the modified IR-illuminated image are schematically illustrated.

[0045] Figure 6 The system schematically illustrates a rolling shutter sensor and an IR light configured to be opened and closed.

[0046] Figure 7 A method for performing background light reduction in an IR-illuminated image depicting a scene is illustrated schematically. Detailed Implementation

[0047] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. The systems, devices, and modules disclosed herein will be described during operation.

[0048] This disclosure relates to the field of background light reduction.

[0049] As discussed above, in situations where visual information is insufficient, such as at night or under other low-light conditions, it is advantageous to use IR images to detect and analyze objects in a scene. However, when the scene depicted in the IR image includes one or more strong light sources (such as streetlights, car headlights, etc.), these strong light sources may cause scattering and reflection that obscures the objects to be detected, making it difficult for analysis software to locate and analyze objects in the IR image. This disclosure aims to provide a method, system, apparatus, and software for background light reduction in such situations, particularly when the objects to be detected are moving within the scene or when the camera capturing the scene is moving.

[0050] Figure 1 This schematically illustrates a low-light scenario involving a vehicle with headlights and license plates. Figure 1 In this process, IR illumination image 102 and non-IR illumination image 104 have been created, as described above and below in further combination. Figures 2 to 7 The subject of discussion. For example... Figure 1 As can be seen, the headlights in the IR illumination image 102 reduce the visibility of the license plate, for example, due to scattering and / or reflection, which is caused by the headlights shining directly into the camera capturing the IR illumination image 102. Advantageously, pixel values ​​from the non-IR illumination image 104 can be used to reduce interfering light in the IR illumination image 102, making it easier to detect and analyze the license plate. Figure 1 The image shows a modified IR illumination image 106, in which the background light in the IR illumination image 102 has been reduced using pixel values ​​from the non-IR illumination image 104. In the modified IR illumination image 106, the license plate is clearly detectable and can be analyzed to detect the registration identifier (AXIS1), which is an alphanumeric ID that uniquely identifies the vehicle or its owner.

[0051] In some embodiments, pixel values ​​from the entire non-IR illumination image 104 (e.g., all pixels in the non-IR illumination image 104) are used to subtract background light from the IR illumination image. In other embodiments, a subset of pixel coordinates is determined from the non-IR illumination image such that when the modified IR illumination image 106 is created, background light subtraction is used only for the corresponding subset of pixels in the IR illumination image 102. The subset of pixel coordinates in the non-IR illumination image 104 can be determined based on an analysis of the luminance of pixel values ​​in the non-IR illumination image 104, such that if the luminance of a pixel exceeds a threshold, the coordinates of that pixel are included in the subset of pixel coordinates. Limiting the set of pixels used for background subtraction can reduce noise added to the modified IR illumination image 104 due to the subtraction operation. Figure 1The example illustrates the concept of selecting a subset of pixel coordinates for reduction, where the coordinates of pixel data 108, 110 corresponding to the headlights and glare from the headlights in the non-IR illumination image 104 can be included in this subset. It should be noted that a subset of pixel coordinates 108, 110 is used as an example to facilitate explanation of the concept of selecting pixel coordinates based on photometric values.

[0052] In some examples, pixel values ​​can be weighted with weight values ​​(e.g., between 0 and 1, such as 0.1, 0.5, 0.9, etc.) before being used for background light reduction. Subtracting a portion (e.g., 10%, 50%, 90%, etc.) of the background light in the non-IR illuminated image 104 can result in less noise being added to the modified IR illuminated image compared to subtracting the full pixel values ​​of the non-IR illuminated image.

[0053] In some embodiments, the same color space is used to capture both the non-IR lighting image 104 and the IR lighting image 102. In these cases, subtraction can be performed directly using element-wise arithmetic (e.g., by subtracting the RGB values ​​of the pixels in the non-IR lighting image 104 from the RGB values ​​of the corresponding pixels in the IR lighting image 102). In other embodiments, one or both of the non-IR lighting image 104 and the IR lighting image 102 are converted to another color space, such as the LAB color space, before performing subtraction, so that the L channel of the non-IR lighting image 104 is subtracted from the L channel of the IR lighting image 102. The resulting modified IR lighting image 106 may or may not be converted back to the original color space of the IR lighting image 102. When capturing the non-IR lighting image 104 and the IR lighting image 102 using different color spaces, subtraction can be achieved by converting one or both to a different color space before subtraction. For example, if the IR lighting image 104 is a grayscale image and the non-IR lighting image 104 is an RGB image, the following steps can be performed:

[0054] 1. Convert the RGB image 104 to the LAB color space, which separates the image's luminance (L) from its color (a and b) dimensions.

[0055] 2. Extract the L channel containing photometric information from the LAB image.

[0056] 3. Subtract the L channel value from the pixel values ​​of the grayscale image 104 using element-wise arithmetic.

[0057] Any other appropriate transformation and / or scaling of one or both of the non-IR illumination image 104 and the IR illumination image 102 can be performed before the reduction is performed.

[0058] In order to enable the use of non-IR illumination image 104 and IR illumination image 102 for subtraction in a scenario where the vehicle is moving while being captured by the camera, or in a scenario where the camera is moving while capturing an image, it is advantageous to capture non-IR illumination image 104 and IR illumination image 102 in close temporal proximity. This can be achieved using a rolling shutter image sensor, which supports capturing two or more images with a smaller time difference compared to the readout time of the entire sensor (e.g., capturing non-IR illumination image 104 first and then capturing IR illumination image 102, or vice versa). Rolling shutter image sensors support staggered configurations, which will now be combined... Figure 2 To describe this staggered configuration.

[0059] Figure 2 The illustration schematically shows the use of a rolling shutter image sensor and a modulated IR light source to capture two images that are staggered in time.

[0060] The rolling shutter image sensor is configured to read out pixel data one pixel line at a time during image capture. Therefore, by modulating the IR light source so that the IR light source changes its on-off state 206 multiple times while the image is captured by the rolling shutter image sensor, each frame in the image frame will include the first group of pixel lines 214 (in...). Figure 2 (The middle is dark) and the second group of pixel lines 216 (in Figure 2 (The middle part is white), the first set of pixel lines 214 includes image data captured when the IR light is on, and the second set of pixel lines 216 includes image data captured when the IR light is off. Advantageously, an IR illumination image can be created at least in part based on the first set of lines 214 of the two image frames 202, 204. Figure 1 Reference numeral 102 in the figures shows that a non-IR illuminated image 104 can be created, at least in part, based on the second set of lines 216 of the two image frames 202, 204. Since the rolling shutter readout pattern combines with the flickering IR light, both images 202, 204 contain patterns of pixel lines (bright stripes) captured when the IR light is off and patterns of pixel lines (dark stripes) captured when the IR light is on. This is achieved by setting the time difference 208 between the first image frame 202 and the second image frame 204 to a time span t corresponding to the on-off cycle of the IR light. span If approximately half of 210 is the same or a multiple thereof, the patterns in the two image frames 202 and 204 will be out of sync. In other words, the first pixel line 212 of the first image frame 202 will be read out at the first time point t, and the corresponding first pixel line 212 of the second image frame 204 will be read out at the second time point t. It was read aloud. Figure 2In this example, a square wave 206 is used to modulate (turn on / off) the IR light source. Half a period of the square wave 206 corresponds to the time difference 208 between two frames. In the example, t span 210 can be at least twice the exposure time of any pixel line in two or more image frames.

[0061] In the example scenario, the on-off period t of the IR light span Between 1ms and 2ms, the shutter time (exposure time) of the rolling shutter image sensor is 0.5ms, and the line time (the time difference between the start of exposure of adjacent pixel rows / lines) is 0.015ms. Other configurations are also feasible. A shorter line time is advantageous because it results in less distortion of moving objects in each image frame 202, 204 captured by the rolling shutter image sensor. However, since the shutter time is not infinitely short, and the line time is typically shorter than the shutter time, some pixels in the pixel line will be exposed while the IR light is on and off. This is... Figure 4 The following diagram illustrates the process. In both the first image frame 202 and the second image frame 204, the pixel lines read out during time span 402 are exposed with the IR light 206 on and off. The number of pixel lines read out during time span 402 corresponds to the ratio between shutter speed and line time. In both the first image frame 202 and the second image frame 204, the pixel lines read out during time span 404 are exposed only when the IR light is on or off.

[0062] Therefore, as Figure 4 As illustrated in the schematic map of “Result” 406, the IR-illuminated image (or non-IR-illuminated image) created from the two captured image frames 202 and 204 will include lines of pixels exposed with IR light on and off (e.g., during time span 402). This may still be sufficient to combine... Figure 1 Subtraction is performed as described to create a modified IR illumination image. However, in situations such as Figure 2 and Figure 4 In the two image settings shown, image data stripes will appear in the modified IR illumination image, which may reduce the quality of image analysis, for example, the modified IR illumination image.

[0063] This problem can be solved using a three-image setup, which will now be combined Figure 3 and Figure 5The three-image setup allows the modulation frequency of the IR light to be matched with the time difference between the first, second, and third image frames, such that one of the three image frames always has 100% IR light on in each row of the image. Correspondingly, one of the three image frames has 100% IR light off in each row of the image.

[0064] Figure 3 The illustration schematically demonstrates the use of a rolling shutter image sensor and a modulated IR light source to capture three temporally staggered image frames 302, 304, and 306. Therefore, the rolling shutter image sensor supports a three-frame staggered mode.

[0065] Combined with the above Figure 2 Similarly described, the IR light source is modulated so that while the image sensor is capturing an image using a rolling shutter, the IR light source changes its on-off state 306 multiple times. Each frame in the image frame will include the first group of pixel lines 214 (in... Figure 3 (The middle is dark) and the second group of pixel lines 216 (in Figure 3 (The middle part is white), the first set of pixel lines 214 includes image data captured when the IR light is on, and the second set of pixel lines 216 includes image data captured when the IR light is off. Advantageously, an IR illumination image can be created at least in part based on the first set of lines 214 of the three image frames 302, 304, and 306. Figure 1 (Ref. 102 in the accompanying drawings), and a non-IR illumination image 104 can be created at least in part based on the second set of lines 216 of the three image frames 302, 304, 306. Figure 1 (Figure reference 104).

[0066] By setting the time difference 308 between the first image 302 and the second image 304, and between the second image 304 and the third image 306, to the time span t of the IR light's on-off cycle. span_2 Approximately one-third of 310, or multiples thereof, can prevent any pixel lines from being exposed when IR light is on and off in both non-IR and IR illuminated images. In other words, in Figure 3 In this embodiment, the rolling shutter image sensor reads out the first pixel line 212 at a first time point t for the first image frame 302. For the second image frame 304, the rolling shutter image sensor... Reading out the first pixel line 212, and for the third image frame 306, the rolling shutter image sensor in Read the first pixel line 212. Then repeat this process for each pixel line in the three image frames 302, 304, and 306. In some examples, t span_2It can be at least 6 times the shutter speed to avoid the pixel lines in each image frame being exposed when the IR light 306 is on and off.

[0067] The result is Figure 5 The image is shown schematically. Combined with the above... Figure 4 Similarly, as described, some pixel lines in each image frame are exposed when the IR light 306 is on and off. However, by capturing three image frames 302, 304, and 306 with the time difference between the first image 302 and the second image 304, and between the second image 304 and the third image 306, being approximately one-third or a multiple of the time span of the on-off cycle of the IR light 306, these pixel lines can be avoided in the result 506. Result 506 thus represents an IR-illuminated image (or a non-IR-illuminated image) created from the three captured image frames 302, 304, and 306, and may not include any pixel lines exposed when the IR light is on and off.

[0068] Figure 6 A system 602 suitable for performing background light reduction in an IR-illuminated image depicting a scene is illustrated schematically by way of example. It will now be combined with... Figure 7 To describe the functionality of the system.

[0069] System 602 includes a rolling shutter image sensor 606 with multiple pixel lines, wherein the rolling shutter image sensor reads pixel data from one pixel line at a time during image capture. Therefore, (S702) the rolling shutter image sensor 606 is provided.

[0070] System 602 further includes an IR light source 604, which is configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate the scene. Thus, (S704) IR light is provided. IR is electromagnetic radiation (EMR) with a wavelength longer than that of visible light. Therefore, it is invisible to the human eye. IR is generally understood to include wavelengths from about 1 mm (300 GHz) to the nominal red edge of the visible spectrum (about 700 nm (430 THz)). Infrared light sources can be used to increase the available ambient light for conversion by night vision devices, thereby increasing visibility in the dark without actually using a visible light source. In some embodiments, the IR light source is used only when the luminance value of natural light in the determined scene does not exceed a threshold luminance, and thus is used... Figure 7Method 700 is shown below. Any type of light sensor (e.g., a light meter, illuminance meter, or photometer) can be used to determine the luminance value of natural light in a scene. The threshold luminance can be configured based on the use of the rolling shutter image sensor and its parameters. The threshold luminance can be set to, for example, 2 lux, 1 lux, 0.5 lux, etc.

[0071] Any suitable IR light source, such as a bulb, lamp, or diode, can be used 604.

[0072] System 602 further includes an IR light control unit 608, which is configured to change the on-off state of the IR light multiple times (S706) while capturing an image with a rolling shutter image sensor.

[0073] The rolling shutter image sensor 606 is configured to capture (S708) two or more image frames (e.g. Figures 2 to 3 (As illustrated in the example), the rolling shutter image sensor is configured to read out image data for two or more image frames that are staggered in time. Therefore, each image frame will include a first set of pixel lines and a second set of pixel lines, the first set including image data captured when the IR light is on, and the second set including image data captured when the IR light is off.

[0074] System 602 further includes an image stitching unit 610, configured to create (S710) (e.g., crop and stitch) an IR illumination image based at least partially on a first set of lines from two or more image frames, and to create (S712) a non-IR illumination image based at least partially on a second set of lines from two or more image frames, such as... Figures 4 to 5 As illustrated in the example, the image stitching unit 610 can be further configured to subtract (S714) the background light from the IR illumination image using pixel values ​​from the non-IR illumination image, thereby creating a modified IR illumination image (such as...). Figure 1 (As shown by reference numeral 106 in the accompanying drawing).

[0075] In some embodiments, system 602 further includes an image analysis unit 612 configured to detect license plates in a modified IR-illuminated image. A rolling shutter image sensor may be provided in a camera 600 used to capture a scene including a road.

[0076] In some embodiments, system 602 further includes a wireless transmitter 614. The wireless transmitter 614 can be used to transmit data such as a modified IR illumination image or any detected license plate, or other data derived from analysis of the modified IR illumination image. The data can be transmitted for storage or further analysis at a server. The data can be received by a human operator trained to take action on the data.

[0077] In the example, system 602 can be implemented in a single device such as camera 600. In other examples, some or all of the different components (modules, units, etc.) 606, 608, 610, 612, 614 can be implemented in a server or the cloud. Typically, the device (camera, server, etc.) used to implement components 606, 608, 610, 612, 614 can include circuitry configured to implement components 606, 608, 610, 612, 614 and more specifically to perform their functions. Advantageously, the features in the described system 602 can be implemented as one or more computer programs executable on a programmable system, including at least one programmable processor (coupled to receive data and instructions from and transmit data and instructions to a data storage system), at least one input device (such as a camera), and at least one output device (such as a display). For example, suitable processors for executing instruction programs include general-purpose microprocessors and special-purpose microprocessors, as well as a single processor or one of a multiprocessor or multicore processor in any type of computer. The processor can be supplemented by or incorporated into an ASIC (Application-Specific Integrated Circuit).

[0078] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the invention are conceivable. For example, the image analysis unit 612 may be configured to detect road signs. The output from the image analysis unit may be used to control a vehicle in which the camera 600 is mounted. It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment or any combination of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the claims.

Claims

1. A method for performing background light reduction in an infrared (IR) illumination image depicting a scene, the method comprising: A rolling shutter image sensor with multiple pixel lines is provided, wherein the rolling shutter image sensor reads pixel data from one pixel line at a time during image capture; An IR light source is provided that is configured to be turned on and off, wherein when turned on, the IR light source is configured to illuminate the scene; While capturing images using the rolling shutter image sensor, the on-off state of the IR light source is changed multiple times; The rolling shutter image sensor captures three image frames in chronological order, including a first image frame and a last image frame in chronological order. The rolling shutter image sensor is configured to read out image data for the three temporally staggered image frames, such that the first pixel line is read out for the last image frame before the last pixel line is read out for the first image frame. Each image frame includes: The first group of pixel lines is exposed only when the IR light source is turned on; The second group of pixel lines is exposed only when the IR light source is turned off; The third group of pixel lines is exposed when the IR light source is turned on and off; The IR illumination image is created using only the first set of pixel lines from the three image frames; A non-IR illuminated image is created using only the second set of pixel lines from the three image frames; The modified IR illumination image is created by subtracting background light from the IR illumination image using pixel values ​​from the non-IR illumination image. The step of changing the on-off state of the IR light source includes: during the second time span t span_2 The IR light source completes its on-off cycle during this period. The IR light source is turned on during the first half of the second time span and turned off during the second half of the second time span. The steps for capturing three image frames include: Capture the first image frame, the second image frame, and the third image frame; The first pixel line is read out for the first image frame at the first time point t using the rolling shutter image sensor. Through the rolling shutter image sensor, in Read the first pixel line for the second image frame; and Through the rolling shutter image sensor, in For the third image frame, the first pixel line is read out. Where x and y are positive natural numbers.

2. The method according to claim 1, wherein, The reduction steps include: For each pixel coordinate in at least one subset of pixel coordinates in the non-IR illumination image, the pixel value at that pixel coordinate in the non-IR illumination image is subtracted from the pixel value at the corresponding pixel coordinate in the IR illumination image.

3. The method according to claim 2, further comprising: The step of selecting the subset of pixel coordinates based on the luminance of pixel values ​​in the non-IR illuminated image.

4. The method according to claim 1, wherein, The reduction steps include: For each pixel coordinate in at least one subset of pixel coordinates in the non-IR illumination image, the pixel value at that pixel coordinate in the non-IR illumination image is multiplied by a weight value, and the weighted pixel value at that pixel coordinate in the non-IR illumination image is subtracted from the pixel value at the corresponding pixel coordinate in the IR illumination image.

5. The method according to claim 1, comprising: An initial step for determining the luminance value of natural light in the scene, wherein the method is terminated when the luminance value exceeds a threshold luminance, and the method continues when the luminance value does not exceed the threshold luminance.

6. The method according to claim 1, wherein, The rolling shutter image sensor is configured to have the same exposure time for all pixel lines.

7. The method of claim 1, further comprising: The step of detecting license plates in the modified IR illumination image.

8. The method according to claim 1, wherein, The rolling shutter image sensor is provided in a camera that captures scenes including roads.

9. The method of claim 1, wherein x = 1 and y = 1.

10. A system for performing background light reduction in an infrared (IR) illumination image depicting a scene, comprising: A rolling shutter image sensor having multiple pixel lines, wherein the rolling shutter image sensor reads pixel data from one pixel line at a time during the capture of an image of the scene; An IR light source is configured to be turned on and off, wherein when turned on, the IR light source is configured to illuminate the scene; The IR light control unit is configured to change the on-off state of the IR light source multiple times while capturing an image using the rolling shutter image sensor; The rolling shutter image sensor is configured to capture three image frames, which are captured sequentially in time and include a first image frame and a last image frame in time. The rolling shutter image sensor is configured to read out image data for the three temporally staggered image frames, such that the first pixel line is read out for the last image frame before the last pixel line is read out for the first image frame. Each image frame includes: The first group of pixel lines is exposed only when the IR light source is turned on; The second group of pixel lines is exposed only when the IR light source is turned off; The third group of pixel lines is exposed when the IR light source is turned on and off; The system further includes an image stitching unit, which is configured as follows: An IR illumination image is created using only the first set of pixel lines from the three image frames; A non-IR illuminated image is created using only the second set of pixel lines from the three image frames; The background light is subtracted from the IR illumination image using pixel values ​​from the non-IR illumination image, thereby creating a modified IR illumination image; The IR optical control unit is configured to: in the second time span t span_2 The IR light source is turned on and off during the period, wherein the IR light source is turned on during the first half of the second time span and turned off during the second half of the second time span; The rolling shutter image sensor is configured as follows: Capture the first image frame, the second image frame, and the third image frame; At the first time point t, the first pixel line is read out for the first image frame; exist Read the first pixel line for the second image frame; and exist For the third image frame, the first pixel line is read out. Where x and y are positive natural numbers.

11. The system of claim 10, further comprising: An image analysis unit is configured to detect license plates in the modified IR illumination image.

12. A non-transitory computer-readable storage medium having instructions thereon that, when executed on a device with processing capabilities, implement the method of claim 1.

Images