Vehicle occupant monitoring system including image acquisition device with rolling shutter image sensor

Abstract

A vehicle occupant monitoring system (OMS) includes an image acquisition device having a rolling shutter image sensor configured to selectively operate in any of: a full resolution image mode in which image frames corresponding to the full image sensor are provided; or a Region of Interest (ROI) mode in which an image frame corresponding to a limited portion of the image sensor is provided. The object detector is configured to receive a full resolution image from the image sensor and identify a ROI corresponding to an object of interest within the image. The controller is configured to obtain an image corresponding to the ROI from the image sensor operating in the ROI mode, the image having an exposure time long enough to cause all rows of the ROI to be illuminated by a common light pulse from the at least one infrared light source and short enough to limit motion blur within the image.

Description

Technical Field

[0001] The present invention relates to a vehicle occupant monitoring system (OMS), which comprises an image acquisition device having a rolling shutter image sensor. Background Art

[0002] Referring now to FIG. 1 , an exposure cycle for capturing an image frame using a conventional global shutter image sensor is shown. The cycle includes a number of phases, in the first phase 10-A, in response to a trigger signal for the start of a frame, all rows of the image sensor are reset simultaneously. In the next phase 10-B, all rows of the image sensor are exposed simultaneously at a sensor gain G for an exposure time T. At the end of the exposure time T, charge is transferred from each sensor photodiode to a corresponding capacitor, i.e., phase 10-C. During the exposure phase, if desired, a light source (e.g., an LED flash) may be used to illuminate the scene for a sufficient time T. f and intensity to illuminate the scene being imaged to achieve a properly exposed image.

[0003] Then, the charges are read sequentially from the multiple row groups 1 ... n of the image. The time from triggering the reset of the row to reading the value from the last row group 10-Dn determines the frame rate.

[0004] In contrast, for a rolling shutter sensor, phases 10A to 10D are performed in a staggered manner for each row group 1 . . . n.

[0005] In a rolling shutter sensor, for the same pixel pitch, the fill factor of the sensor pixels is better and more of the sensor area is exposed to light. For the same pixel pitch, a rolling shutter sensor has a better dynamic range due to the larger well capacity. Rolling shutter sensors are also easier and cheaper to manufacture.

[0006] In a vehicle driver monitoring system (DMS), a full HD (1920×1080) camera sensor and a lens assembly with a 60-70 degree field of view are required. Such an imaging system can produce an image of the driver's facial area at a distance of 1 meter, which reaches approximately 160×160 pixels. For an occupant monitoring system (OMS) to monitor front seat occupants and possibly rear seat occupants, a wide field of view lens assembly with a horizontal field of view of 120 degrees or more may be required. In either case, the image sensors are generally required to be sensitive to infrared (IR) wavelengths, but also to visible wavelengths, so that they operate in both day and night conditions.

[0007] However, rolling shutter sensors are known to be susceptible to distortion issues when the sensor or scene / object moves due to the staggered exposure times of the rows within the sensor. As a result, they are not typically used in surveillance applications where detection and classification of subject movement is important.

[0008] Furthermore, for a vehicle occupant monitoring system, ideally the vehicle cabin would be illuminated with an infrared (IR) light source operating in a pulsed mode, which prevents heat buildup within the camera housing during prolonged monitoring periods. (Note that when such a camera is operated in daylight, color mode, artificial lighting is not used, as this might be distracting to the driver or other occupants.) However, as will be clear from FIG. 1 , in a rolling shutter sensor, ensuring that all rows are exposed with a common pulse requires extending the exposure cycle to at least the time from t0 to t1 , which may be longer than the time for subject motion (and even motion due to camera vibration) to cause image blur.

[0009] This has effectively eliminated the use of cameras with rolling shutter sensors from such monitoring applications. Summary of the invention

[0010] According to a first aspect of the present invention, there is provided a vehicle occupant monitoring system (OMS) according to claim 1, the vehicle occupant monitoring system comprising an image acquisition device having a rolling shutter image sensor.

[0011] In an embodiment, occupant monitoring can operate by acquiring images from a region of interest (ROI) of an image sensor with exposure times long enough to allow the scene to be illuminated with a pulsed IR light source, but short enough to avoid subject or acquisition device motion blur.

[0012] According to a second aspect of the present invention, there is provided an OMS according to claim 7, comprising an image acquisition device having a rolling shutter image sensor.

[0013] In an embodiment of the second aspect, the image sensor may switch between a color mode of operation and a monochrome mode of operation in response to changes in ambient light levels.

[0014] According to a third aspect of the present invention, there is provided an OMS according to claim 9, comprising an image acquisition device having a rolling shutter image sensor.

[0015] In an embodiment of the third aspect, exposure parameters of the image sensor when operating in ROI mode may be controlled in an optimal manner.

[0016] According to a fourth aspect, a vehicle occupant monitoring system OMS according to claim 11 is provided.

[0017] In an embodiment of the fourth aspect, illumination of the image may be improved within an image processing pipeline to account for uneven illumination of the cabin by one or more infrared light sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0019] Figure 1 shows the operational differences between a global shutter image sensor and a rolling shutter image sensor;

[0020] Figure 2 The image acquisition equipment within the vehicle occupant monitoring system was demonstrated;

[0021] Figure 3 A rolling shutter image sensor switching between a full resolution mode and a ROI exposure mode according to an embodiment of the present invention is shown;

[0022] Figure 4 An exposure control mechanism according to an embodiment of the present invention is presented for determining whether an image sensor should operate in an RGB mode or a monochrome image acquisition mode when acquiring a full resolution image;

[0023] Figure 5 An exposure control mechanism according to an embodiment of the present invention is presented for determining exposure parameters when acquiring a region of interest (ROI) from an image sensor; and

[0024] Figure 6 Correction for cabin illumination according to an embodiment of the present invention is demonstrated. DETAILED DESCRIPTION

[0025] See now Figure 2 , shows a vehicle cabin including an image acquisition device 104 for a vehicle occupant monitoring system according to an embodiment of the present invention.

[0026] In an embodiment, the image acquisition device 104 is mounted on a solid / rigid surface, such as on a vehicle A-pillar, or in this embodiment, on the dashboard, to avoid rolling shutter effect artifacts caused by vehicle movement. In any case, the field of view 114 of the lens assembly for the device 104 enables the faces of the vehicle occupants 108, 112, or indeed any body part of interest, to be imaged at a high enough resolution that the monitoring application can use the images acquired by the device 104 to properly perform its tasks.

[0027] As indicated, such applications typically require continued operation during low light or nighttime conditions, in which the vehicle cabin is illuminated using one or more infrared (IR) light sources, in this embodiment one or more LEDs 600. Such light sources 600 may be positioned at multiple locations around the cabin to ensure that vehicle occupants can be imaged using IR-sensitive image sensors even in low ambient light conditions.

[0028] Ideally, such a light source 600 is operated in a pulsed mode to avoid heat build-up within the lighting fixture, but as explained previously, this means that the image exposure time is so long that motion blur is introduced in regions of interest (e.g., faces) within the image. This blur means that applications that rely on, for example, recognizing eye gaze may not operate correctly.

[0029] Embodiments of the present invention employ an image acquisition device having a rolling shutter image sensor. The sensor is capable of operating in a region of interest (ROI) mode, in which a ROI within the image sensor can be defined and image information can be obtained only from the portion of a selected row corresponding to the ROI during subsequent image acquisition. An example of such a sensor is available from ON Semiconductor of Phoenix, Arizona, model number AR0820AT.

[0030] See now Figure 3 , the image acquisition device 104 including the sensor 300 operates in two modes. In the first full resolution mode, image information from the entire sensor is obtained. The image can be a multi-plane color image drawn from the RGB sensitive sub-pixels of the image sensor 300, or alternatively, the image can include a monochrome image drawn from the IR sensitive pixels of the sensor 300.

[0031] In some embodiments, this involves using a sensor with sub-pixels that are sensitive to R, G, B, or IR light, respectively, and the sensor switches between two modes: capturing images from the RGB sub-pixels in bright light conditions and capturing images from the IR-sensitive pixels in low light conditions.

[0032] However, in some embodiments of the present invention, only one set of pixels is used. The image sensor 300 is filtered so that each sub-pixel is sensitive to R+IR, G+IR, or B+IR light, respectively. Under sufficient light conditions, multi-plane image information is collected separately from the R+IR, G+IR, and B+IR sub-pixels. It should be understood that this may result in some light leakage and loss of image quality compared to other visible light image sensors in which RGB pixels are filtered to filter out IR light. However, the full resolution image collected in this manner will be of high enough quality to be used for typical occupant monitoring system (OMS) 310 applications.

[0033] On the other hand, in low light conditions, the sensor 300 can be switched so that it aggregates information from the R+IR, G+IR, and B+IR sub-pixels to produce a high-quality, high-resolution monochrome image. It will be appreciated that this allows the sensor to operate at light levels several times lower than the RGB mode. If the light level drops further, the IR light source 600 can be actuated, but it will be appreciated that this may require a longer exposure time so that a pulse illuminates each row of the image. Nevertheless, doing so will highlight the IR components of the scene, which are particularly useful for driver monitoring systems (DMS).

[0034] Note that in some embodiments, image sensor 300 may include a Bayer array, where color plane information for a given location in an image may be determined by interpolation from sub-pixels surrounding the location. Thus, there may not be a one-to-one correspondence between sub-pixels and image locations - this is particularly true when image resolution changes.

[0035] In this patent specification, the term subpixel is used for a single well within an image sensor, which, as described above, can be sensitive to R, G, B, IR, or R+IR, G+IR, B+IR light. In fact, in some image sensors, there can be subpixels that are broadly sensitive to visible light or RGB+IR light.

[0036] Thus, when the light level in the cabin is sufficient, an RGB image can be obtained without any artificial illumination of the cabin. On the other hand, if the light level is too low to obtain a useful RGB image, an IR image can be obtained. Due to the low light level, the image may require a longer exposure time, and if a pulsed flash from an IR light source is required, the pulsed flash must illuminate all rows of the image sensor for a given frame during its exposure time and therefore also require a longer exposure time. This may mean that such image frames are prone to a certain degree of blur, however, downstream applications that analyze features of such images do not typically rely on such images being completely sharp. One such example is the OMS 310, which may typically involve, for example, recognizing the general posture of a vehicle occupant to determine whether the occupant is correctly and safely seated. (Where the OMS is based on a neural network, the training data for the network may be augmented with motion blurred images.)

[0037] In any case, in embodiments of the present invention, the full resolution image obtained from the sensor 300 may also be analyzed to determine a specific region of interest (ROI). In the example, the ROI is a facial region around the driver of the vehicle, such that the driver may be monitored by the DMS 320. As such, the face detector 330 is applied to the full resolution image to identify the facial region.

[0038] The image sensor 300 can now be controlled to operate in ROI mode, in which only image information from a limited portion of a limited number of rows of the sensor is required. Figure 3 As can be seen in Figure 3, even though the driver's relatively large face is within the camera's field of view, it only extends across less than one-third of the full sensor row. This means that the exposure time for a rolling shutter image does not have to be as long as the exposure time to expose all rows of an image under illumination from a single pulse (or a limited number of pulses) from one or more light sources. This allows a clear, well-exposed image of the subject to be acquired for analysis by the DMS 320.

[0039] In some embodiments of the present invention, an image of the ROI is captured for each full resolution image, and an image package including the pair of full resolution images and the ROI image is provided for analysis by downstream applications. However, it should be understood that this need not be the case, for example, if DMS is a preferred option, full resolution images need only be acquired when a new position of the ROI is required. For example, the DMS 320 may fail to detect facial features within the image of the ROI due to movement of the driver's face within the camera's field of view, and therefore the face detector 330 may need to acquire and analyze full resolution images to obtain a new position of the ROI that defines the driver's face.

[0040] As noted above, the present invention is not limited to ROIs that include facial regions, and in other embodiments it may be necessary to define an ROI of the driver or occupant's body, for example, to perform gesture detection and recognition.

[0041] It will be appreciated that such an ROI may extend over a proportionally larger number of horizontal rows than the facial area, and thus in such an application the sensor 300 may need to be rotated 90° so that the rows extend upwards. Again, this would allow the acquisition of an ROI image of a single occupant using only a fraction of the rows required for a full resolution image, and thus allow a clear image to be acquired with a short exposure time under illumination by a single (or limited number of) pulses from the light source.

[0042] See now Figure 4 As previously mentioned, where the sensor includes sub-pixels sensitive to R+IR, G+IR, and B+IR light, the exposure controller 700 can cause the image sensor 300 to switch between two modes: operating in a multi-plane RGB mode or operating in an aggregated monochrome IR mode.

[0043] If the sensor is operating in RGB mode, the light level in the cabin may drop to a point where the exposure needs to be increased (step 40) to provide a properly exposed image, such as an image with a minimum number of pixels that are fully exposed (cropped) without being saturated, thereby providing maximum contrast within the image.

[0044] If this is accomplished by increasing the exposure time below a threshold where subject movement may cause motion blur or by increasing the gain below a maximum gain level, the exposure controller 700 determines that the increased exposure level may be maintained in RGB mode (step 42).

[0045] On the other hand, if neither a further increase in gain nor a further increase in exposure time is permitted, the exposure controller 700 determines that it is no longer possible to remain in the RGB mode and will switch to the monochrome mode (step 44).

[0046] While in monochrome mode, exposure controller 700 may detect that light levels are still decreasing (step 46), and may increase exposure time or gain in this case as well (step 48).

[0047] On the other hand, if an increase in ambient light levels is detected at step 46, the exposure controller 700 will first determine whether these ambient light levels have increased to the point where a properly exposed image can be acquired in RGB mode. If not, the exposure controller 700 will simply reduce the exposure time or gain level based on the system's priorities (step 50). Thus, if the exposure time is relatively long, the exposure controller 700 may tend to reduce the exposure time first to avoid blurring, and then reduce the gain to limit noise.

[0048] In any case, if the ambient light level has increased sufficiently, the exposure controller 700 can switch back to RGB mode to capture the next fully exposed image (step 52). This may require an increased exposure time and / or a step change in gain, since the sub-pixel values ​​are now partitioned into different color planes rather than summing their outputs. Nevertheless, if the ambient light level continues to increase once in RGB mode, the exposure time and / or gain can be reduced (step 54).

[0049] Because of the ability to switch to operating in monochrome full-resolution mode, exposure times can still be kept short enough that subject motion is not a critical issue in such images.

[0050] On the other hand, when operating in ROI mode, it is generally desirable to obtain IR-illuminated monochrome images out of preference. Ideally, these images will be acquired with the shortest possible exposure time and the lowest possible gain to reduce both motion blur and noise. Nevertheless, it is desirable to be able to illuminate the subject with a single pulse (or a small number of pulses) from an IR light source 600 (e.g., one or more LEDs arranged around the vehicle cabin), and therefore it is desirable to control the source to ensure a properly illuminated image.

[0051] See now Figure 5, when operating in ROI monochrome mode, the exposure controller 700 needs to determine the appropriate exposure parameters for the ROI image. Note that this will be completely independent of the exposure time and gain determined for the full resolution image, especially since the IR illumination of the image using the LED IR source 600 needs to be considered. The process begins with the LED IR source 600 turned off. To capture any given image, the exposure controller 700 will determine whether the ambient light level requires an increase or decrease in the exposure level (step 60). If the ambient conditions are darker than the previously captured ROI image, then the exposure level needs to be increased. The exposure controller 700 first determines whether this can be accomplished by increasing the exposure time. As should be understood from Figure 1, in a rolling shutter image sensor, if the exposure time is less than the threshold time required to read out all rows of the image, then all rows of the image cannot be illuminated with a single pulse from the LED IR light source. Assuming that in ROI mode, the number of selected rows will be small enough so that the exposure time required to allow the threshold level of exposure required to expose all rows of the ROI with a single pulse is still short enough not to cause motion blur issues. Therefore, when the exposure time does not exceed the threshold, the exposure level is increased by increasing the exposure time (step 62).

[0052] On the other hand, once the exposure time has exceeded this threshold, the exposure time should not be increased further, as this would start to introduce motion blur into the acquired image. Therefore, the next option is to increase the LED intensity (step 64). If the LED intensity cannot be increased further, the sensor gain is increased (step 66). Obviously, once the gain has been increased to its maximum value, it will not be possible to increase the exposure level further by this parameter. Once an IR light source 600 is employed that is strong enough to illuminate an image of an ROI of any size within the cabin under dark ambient conditions, this should not occur, but obviously there may be abnormal situations, such as when one or more of the IR light sources 600 are blocked.

[0053] look Figure 5 On the other side, as ambient light conditions increase, the first goal is to reduce the exposure level by reducing the gain (step 68). Once the gain has been reduced below a threshold amount (1 in this embodiment), the exposure controller 700 next seeks to reduce the LED IR light source intensity (step 70). Only when the LED is no longer needed can the exposure time be reduced, perhaps until the exposure time is shorter than a threshold time required to use the IR light source 600 (step 72).

[0054] As described above, one or more LED IR light sources 600 may be required to properly illuminate the occupants of the vehicle so that they can be monitored by the OMS 310 in general, or the driver in particular, by the DMS 320. In practice, it is not only necessary to monitor the front seat occupants of the vehicle, but also the rear passengers. Figure 6 In one example of the present application, a pair of LED IR light sources 600-A, 600-B are mounted on the ceiling of the cabin toward the center of the cabin. The LEDs face outward so that they illuminate both the front seat occupants and the rear passengers as far as possible from the front and sides. Nevertheless, it will be understood that this lighting design, and indeed various other lighting designs, tend to illuminate the cabin unevenly, as shown by the two frustoconical illumination areas 610-A, 610-B produced by the sources 600-A, 600-B.

[0055] While it would be desirable to produce completely uniform illumination of all potential vehicle occupants in all circumstances, this may involve providing a greater number of IR light sources, thereby increasing system complexity and expense.

[0056] Some embodiments of the invention take advantage of the fact that LED IR light sources 600-A, 600-B are typically the only IR light sources in the cabin (if used at all), and since the layout of the cabin (whether seats are occupied or not) is typically similar, an illumination map of the cabin can be generated. One way to generate such a map is to simply image the cabin in otherwise (IR) dark light conditions (where the only IR light sources in the cabin are light sources 600-A, 600-B) and store it in a lookup table. For example, from Figure 5 As will be appreciated in the drawings, the light sources may be operated at a range of intensities, and in such cases, an illumination map may be obtained at a plurality of preset light source intensities. Interpolation may then be performed to generate an appropriate illumination map if intensities intermediate these intensities are employed. Alternatively, the map information may be converted into a function that, when executed, may generate illumination information corresponding to the illumination pattern of light sources 600-A and 600-B at any given intensity.

[0057] In either case, the illumination map may be stored and then made available for processing in the image processing pipeline (IPP) 800 ( Figure 3 ), which is associated with the image sensor 104 to pre-process images captured by the image sensor 104. One task commonly performed within the IPP 800 is lens shading correction, wherein the IPP 800 processes the raw image to compensate for non-uniformities in the transmittance of the light path through the image capture device lens assembly, thereby improving the uniformity of illumination and color within the captured image.

[0058] The lens shading correction is based on a lens shading function or map having a field of view covering the device 104, thereby allowing correction of illumination values ​​of the plane(s) of the acquired image.

[0059] By combining the lens shading function / map with the illumination map of the cabin, when the LED IR light source is applied (and considering the selected intensity of the light source, as for Figure 5 As discussed above, the lens shading correction stage of the IPP 800 can produce a processed image that appears to be more uniformly illuminated in the area 620 of the field of view of the device 104 (which normally includes the occupants of the vehicle).

Claims

1. A vehicle occupant monitoring system OMS, comprising: An image acquisition device having a rolling shutter image sensor, the device being configured to selectively operate in any of the following modes: a full resolution image mode in which image frames corresponding to the full image sensor are provided; or a region of interest ROI mode, wherein an image frame corresponding to a limited portion of a limited number of rows of said image sensor is provided; at least one infrared (IR) light source disposed within a cabin of the vehicle and configured to illuminate at least one occupant of the vehicle; an object detector configured to receive a full resolution image from the image sensor and identify a ROI corresponding to an object of interest within the image; as well as A controller is configured to obtain an image corresponding to the ROI from the image sensor operating in ROI mode, the image having an exposure time long enough to cause all rows of the ROI to be illuminated by a common light pulse from the at least one light source and short enough to limit motion blur within the image.

2. The vehicle occupant monitoring system according to claim 1, wherein: The object includes a face of an occupant.

3. The vehicle occupant monitoring system according to claim 2, wherein: The occupant is the driver of the vehicle.

4. The vehicle occupant monitoring system of claim 1, wherein: The image sensor includes a sub-pixel array, which is selectively sensitive to the following colors of light: red light and infrared light; blue light and infrared light; and green and infrared light, the device being configured to selectively operate in either: a color mode, in which a multi-plane image frame corresponding to the full image sensor is provided, each plane originating from a red, green, or blue sensitive sub-pixel, respectively; or a monochrome mode, in which sensor information from the sub-pixels is aggregated to provide a single image plane; and The controller: in response to detecting that the illumination condition decreases below a first threshold amount while operating in the color mode, switches to the monochrome mode for subsequent image acquisition; in response to detecting that the illumination condition increases above a second threshold amount while operating in the monochrome mode, switches to the color mode for subsequent image acquisition; in response to detecting that the illumination condition decreases below a third threshold amount while operating in the color mode, increases the exposure time or gain for subsequent image acquisition, the third threshold amount being higher than the first threshold amount; and reducing exposure time or gain for subsequent image acquisition in response to detecting an increase in lighting conditions above a fourth threshold amount when operating in the monochrome mode, the fourth threshold amount being lower than the second threshold amount, The second threshold is higher than the first threshold.

5. The vehicle occupant monitoring system of claim 1, wherein: Each pixel of the image sensor includes a pixel sensitive to at least infrared light, and The controller, when operating in the ROI mode: in response to detecting a decrease in lighting conditions, increases an exposure time of a subsequently acquired image unless the exposure time exceeds a threshold time for illuminating a row of the ROI with a single IR light pulse from the IR light source; In response to the exposure time exceeding the threshold time, increasing the intensity of the at least one IR light source unless the increased intensity exceeds a maximum intensity; And in response to the increased intensity exceeding the maximum intensity, increasing the gain of the pixel of the image sensor.

6. The vehicle occupant monitoring system of claim 1, wherein: The image acquisition device includes a lens assembly having a varying transmittance across a field of view of the image sensor; and The at least one infrared (IR) light source is configured to illuminate at least one occupant of the vehicle with illumination that varies across a field of view of the image sensor; The occupant monitoring system further comprises: An image processing pipeline is configured to obtain images acquired from the image sensor and pre-process them according to the lens shading map and the cabin illumination map to compensate for both the varying transmittance and the varying illumination so as to provide a more uniformly illuminated image to a controller for further analysis.

7. A vehicle occupant monitoring system, comprising: An image acquisition device having a rolling shutter image sensor, wherein the rolling shutter image sensor includes an array of sub-pixels selectively sensitive to the following respective colors of light: red light and infrared light; blue light and infrared light; and green and infrared light, the device being configured to selectively operate in either: a color mode, in which a multi-plane image frame corresponding to the full image sensor is provided, each plane originating from a sub-pixel sensitive to red, green, or blue, respectively, or a monochrome mode, in which sensor information from the sub-pixels is aggregated to provide a single image plane; The controller: in response to detecting that the illumination condition decreases below a first threshold amount while operating in the color mode, switches to the monochrome mode for subsequent image acquisition; in response to detecting that the illumination condition increases above a second threshold amount while operating in the monochrome mode, switches to the color mode for subsequent image acquisition; in response to detecting that the illumination condition decreases below a third threshold amount while operating in the color mode, increases the exposure time or gain for subsequent image acquisition, the third threshold amount being higher than the first threshold amount; and reducing exposure time or gain for subsequent image acquisition in response to detecting an increase in lighting conditions above a fourth threshold amount when operating in the monochrome mode, the fourth threshold amount being lower than the second threshold amount, The second threshold is higher than the first threshold.

8. The vehicle occupant monitoring system according to claim 7, further comprising: at least one infrared (IR) light source disposed within a cabin of the vehicle and configured to illuminate at least one occupant of the vehicle; The controller is configured to activate the at least one infrared (IR) light source when operating in the monochromatic mode.

9. A vehicle occupant monitoring system (OMS), comprising: An image acquisition device having a rolling shutter image sensor, the device being configured to selectively operate in any of the following modes: a full resolution image mode in which image frames corresponding to the full image sensor are provided; or a region of interest (ROI) mode, wherein an image frame corresponding to a limited portion of a limited number of rows of said image sensor is provided, said image sensor comprising an array of pixels, each pixel comprising a pixel sensitive to at least infrared light, at least one infrared (IR) light source disposed within a cabin of the vehicle and configured to illuminate at least one occupant of the vehicle; an object detector configured to receive a full resolution image from the image sensor and identify a ROI corresponding to an object of interest within the image; as well as a controller configured to obtain an image corresponding to the ROI from the image sensor operating in the ROI mode, the controller: in response to detecting a decrease in lighting conditions, increasing an exposure time of a subsequently acquired image unless the exposure time exceeds a threshold time for illuminating a row of the ROI with a single IR light pulse from the IR light source; In response to the exposure time exceeding the threshold time, increasing the intensity of the at least one IR light source unless the increased intensity exceeds a maximum intensity; And in response to the increased intensity exceeding the maximum intensity, increasing the gain of the pixel of the image sensor.

10. The vehicle occupant monitoring system according to claim 9, wherein: The controller is configured to: in response to detecting an increase in lighting conditions, reduce the gain of a subsequently acquired image unless the reduced gain is less than a threshold gain; in response to the reduced gain being less than the threshold gain, reduce the intensity of the IR light source unless the reduced intensity is less than 0% intensity; and in response to the reduced intensity being less than 0% intensity, reduce the exposure time of the subsequently acquired image.

11. A vehicle occupant monitoring system, comprising: An image acquisition device, the image acquisition device comprising an image sensor and a lens assembly, the lens assembly having a variable transmittance over a field of view of the image sensor; at least one infrared (IR) light source disposed within a cabin of the vehicle and configured to illuminate at least one occupant of the vehicle with illumination that varies across a field of view of the image sensor; as well as An image processing pipeline is configured to obtain an image acquired from the image sensor and pre-process the image according to a lens shading map and a cabin illumination map to compensate for both the varying transmittance and the varying illumination so as to provide a more uniformly illuminated image to a controller for further analysis.

12. The vehicle occupant monitoring system according to claim 11, wherein: One or both of the lens shading map and the cabin illumination map are obtained in any of the following ways: by executing a function; or by reading pre-calibrated information from a lookup table.

13. The vehicle occupant monitoring system according to claim 11, wherein: The illumination map is a function of the intensity of the light source used to illuminate the image.

14. The vehicle occupant monitoring system according to claim 11, wherein: At least two of the IR light sources are disposed atop the cabin toward the center of the cabin and are positioned outwardly and downwardly toward respective sides of the cabin.

15. The vehicle occupant monitoring system according to claim 11, wherein: The image acquisition device is mounted on one of the A-pillar or the dashboard of the vehicle.

Images