Multispectral image processing system for face detection

Abstract

The present invention discloses an image processing system, which includes at least one image sensor, the at least one image sensor includes a plurality of sub-pixels and is configured to provide a first image plane and a second image plane, the first image plane comes from a first sub-pixel group selectively sensitive to a first NIR light band, and the second image plane comes from a second sub-pixel group selectively sensitive to a second NIR light band. A NIR light source is capable of emitting a first NIR light corresponding to the first NIR light band and a second NIR light corresponding to the second NIR light band, respectively. The system can be configured to operate according to at least a first operating mode, wherein a face detector is configured to detect a first face in at least the first image plane and a second face in the second image plane at a position that does not overlap with the first face in space.

Description

Technical Field

[0001] The present invention relates to a multispectral image processing system for face detection and applications based on such face detection. Background Art

[0002] Real-time face detection and tracking is well known in image processing, for example as described in European Patent No. EP2052347 (Ref: FN-143). These techniques enable one or more facial regions within an imaged scene to be easily delineated and allow subsequent image processing based on this information.

[0003] Such image processing may include facial recognition that attempts to identify the imaged individual; autofocus by bringing detected and / or selected facial regions into focus; or defect detection and / or correction of facial regions.

[0004] Regarding individual identification based on facial features, A.K. Jain, A. Ross, and S. Prabhakar, "An introduction to biometric recognition," IEEE Trans. Circuits Syst. Video Technol., vol. 14, 2004, discloses that the iris of the eye is a nearly ideal biometric. Typically, images of the iris are best acquired in a dedicated imaging system using infrared (IR) illumination, typically near infrared (NIR) over 700 nm.

[0005] The iris region is typically extracted from the identified eye region, and a more detailed analysis may be performed to determine if a valid iris pattern can be detected. For example, J. Daugman, "New methods in iris recognition," IEEE Trans. Syst. Man. Cybern. B. Cybern., vol. 37, pp. 1167-1175, 2007 discloses a series of additional improvements that may be used to determine the exact shape of the iris and eye pupil. It is also common practice to convert the iris from a polar coordinate system to a rectangular coordinate system, but this is not required.

[0006] Detecting and tracking the eye or iris area can also be used to determine gaze or a person's condition, such as fatigue or other health conditions, which is particularly useful in driver monitoring systems (DMS) integrated into vehicles.

[0007] Additionally, most cameras and smartphones can recognize specific patterns such as "winks" and "smiles" of faces that are tracked in real time, and can adjust the timing of main image acquisition to ensure that objects in the scene are in focus and not blinking or smiling, such as disclosed in WO2007 / 106117 (reference: FN-149).

[0008] When acquiring differently illuminated objects, a common problem when capturing images within a scene is that the system has limited dynamic range. Specifically, areas of the acquired image corresponding to bright areas of the scene tend to be overexposed, while areas of the acquired image corresponding to darker areas of the scene tend to be underexposed.

[0009] This problem can particularly affect the acquisition of active illumination of faces within a scene that extend over a significant depth of field within the scene, such as the faces of occupants located at different rows within a vehicle imaged from a camera positioned toward the front of the vehicle (e.g., near a rearview mirror). Specifically, if the exposure is set to acquire a properly exposed image of a face near the camera (more illuminated by the light source), then the acquired image of a face farther from the camera (less illuminated by the light source) tends to be underexposed. Vice versa, if the exposure is set to acquire a properly exposed image of a face farther away, then the image of a closer face tends to be overexposed.

[0010] A known solution for acquiring images with a high dynamic range (HDR) is to capture a sequence of consecutive images of the same scene at different exposure levels, for example by varying the exposure time for acquiring each image, wherein a shorter exposure time is used to properly capture bright scene areas, and a longer exposure time is used to properly capture darker scene areas. The acquired images can then be combined to create a single image in which each area in the scene is properly exposed.

[0011] It is easy to understand how this solution can be applied very satisfactorily to scenes with static objects (such as landscapes), but is impractical for capturing faces that are relatively close to the camera and may move during the acquisition of consecutive images, thus causing artifacts when trying to construct an image of the scene. It should also be noted that it is not possible to acquire such a variable exposure image sequence using a rolling shutter technique.

[0012] By Abbas El Gamal of Stanford University, "High Dynamic Range Image Sensors", ISSCC'02 ( http: / / cafe.stanford.edu / ~abbas / group / papers_ and_pub / isscc02_tutorial.pdf), further disclosed the use of HDR CMOS image sensors with spatially varying pixel sensitivities. Specifically, an array of neutral density (ND) filters is deposited on the image sensor such that in a single captured image of a scene, sensor pixels associated with darker filters can be used to acquire bright areas of the scene, while sensor pixels associated with brighter filters can be used to acquire darker areas of the scene. However, this document does not address face acquisition and detection over a wide depth of field within a scene, particularly when active IR illumination is used. Summary of the invention

[0013] According to an aspect of the present invention, there is provided an image processing system.

[0014] Embodiments of these systems are based on employing an image sensor having a plurality of groups of subpixels, each subpixel being configured to acquire an image plane of a given scene substantially simultaneously in a different NIR light band and with a different sensitivity. The image sensor is employed in conjunction with at least one active NIR illumination source capable of emitting NIR light matching the NIR light band of the corresponding subpixel group.

[0015] Notably, in some embodiments, use of larger, more sensitive groups of sub-pixels in conjunction with matching higher intensity emitted NIR light allows for acquisition of properly exposed images of faces that are farther from the system (which will tend to be poorly illuminated and therefore underexposed due to their greater distance from the NIR illumination source), and use of fewer, less sensitive groups of sub-pixels in conjunction with matching lower intensity emitted NIR light allows for simultaneous acquisition of properly exposed images of faces that are closer to the system (which will tend to be over-illuminated and therefore overexposed due to their closer proximity to the NIR illumination source).

[0016] Thus, this combination of features allows a natural balancing of exposure levels to appropriately acquire faces (ie, with a desired level of facial detail) at different depths into the imaged scene, thereby achieving a greater acquisition dynamic range than is possible with typical single wavelength image processing systems.

[0017] These embodiments are advantageously capable of operating in a first mode of operation for detecting faces at different depths within a scene with increased accuracy, but may also provide advantages when operated according to a second mode of operation for cardiac pulse monitoring.

[0018] In practice, images of a given detected face acquired from groups of sub-pixels of different sensitivities may be used to analyze a differential signal indicative of illumination differences between facial images acquired within a sequence of acquisition cycles.

[0019] In this differential monitoring method, changes in the surface illumination of the monitored face due to factors unrelated to the cardiac pulse repetition frequency (such as changes in ambient lighting or changes in movement of the monitored person) will affect the facial images acquired from different sensitivity sub-pixel groups in the same way, and will therefore tend to cancel each other out in the monitored differential signals.

[0020] In some embodiments, the system can switch from a first operating mode to a second operating mode based on adjustment of acquisition settings to acquire properly exposed images of the same face from different sensitivity sub-pixel groups.

[0021] In still other embodiments, differential pulse rate monitoring may be implemented separately from the ability to properly acquire facial images at different scene depths. In this way, these embodiments may be provided without features specifically designed to compensate for exposure levels of faces at different distances from the system, such as having different numbers of sub-pixels in different sensitivity sub-pixel groups and / or emitting corresponding matching NIR light with different intensities.

[0022] According to other aspects of the present invention, there is provided a portable electronic device or a vehicle occupant monitoring system comprising the system according to the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0024] Figure 1 An image processing system according to an embodiment of the present invention is shown;

[0025] Figure 2 A multi-spectral filter array image sensor according to an embodiment of the present invention is shown having three groups of sub-pixels associated with respective different bandpass filters; and

[0026] Figure 3 Shown with Figure 2 The frequency responses of the bandpass filters associated with the sensor's subpixel groups are shown in , along with the matching LED emission bands and relative pixel sensitivities. DETAILED DESCRIPTION

[0027] Reference now Figure 1 , shows an image processing system 10 according to an embodiment of the present invention.

[0028] The system 10, which may be integrated into a portable device (e.g., a camera, a smartphone, a tablet, etc.) or may be integrated into a vehicle safety system (such as a driver monitoring system (DMS)), includes at least one central processing unit (CPU) 24, which typically runs operating system software as well as general application software. For example, in a portable device, the CPU 24 may run a camera application, a browser, a messaging, email, or other application. In a vehicle safety system, the CPU 24 may run a camera application that is dedicated to monitoring the status of a driver or other occupant, particularly for determining the health or attention of the occupant.

[0029] The operating system may be set up so that a user must authenticate themselves in order to unlock the system and gain access to applications installed on the system; or individual applications running on the system may require the user to authenticate themselves before being able to access sensitive information.

[0030] System 10 also includes at least one NIR illumination source 16 and, in some embodiments, at least one visible light illumination source 17 capable of actively illuminating a scene in front of system 10 , and lens assembly 12 capable of focusing light reflected from the illuminated scene onto image sensor 14 .

[0031] In this embodiment, the sensor 14 is a multi-spectral filter array image sensor that is capable of acquiring multi-band image planes of an illuminated scene substantially simultaneously during an image acquisition cycle. For example, an overview of using a multi-spectral filter array to simultaneously acquire multi-band visible and NIR images and other multi-spectral imaging acquisition techniques is provided in P. Lapray et al., "Multispectral Filter Arrays: Recent Advances and Practical Implementation," Sensors 2014, 14(11), 21626-21659.

[0032] refer to Figure 2 , the sensor 14 includes an array of pixels 100 and is formed by a multi-spectral filter array (MSFA) mounted on or arranged near a common image CMOS sensor to filter incident light before it reaches the CMOS sensor.

[0033] The MSFA is patterned similar to a typical Bayer pattern arrangement and includes for each sensor pixel 100: a spectral bandpass filter for passing a lower NIR light band centered at a wavelength of 875nm to sub-pixel I8, a spectral bandpass filter for passing a higher NIR light band centered at a wavelength of 950nm to sub-pixel I9, and a spectral filter for passing a visible bright red light band centered at a wavelength of 700nm to sub-pixel R.

[0034] And refer to Figure 3 , each pixel 100 includes: two sub-pixels I8 sensitive to the NIR light band passed through the filter with passband 101, one sub-pixel I9 sensitive to the NIR light band passed through the corresponding filter with passband 102, and one sub-pixel R sensitive to the red light band passed through the corresponding filter with passband 105. Therefore, the ratio between the number of sub-pixels I8 and sub-pixels I9 (and between the number of sub-pixels I8 and sub-pixels R) in the array of pixel 100 is 2:1.

[0035] The group of sub-pixel I8, sub-pixel I9 and sub-pixel R provides three sensor channels for acquiring three corresponding image planes of an illumination scene at different wavelength bands substantially simultaneously during each image acquisition cycle.

[0036] As is known, for example, from P. Lapray et al. cited above, the quantum efficiency (QE) of CMOS sensors decreases significantly as the wavelength of the incident radiation increases.

[0037] like Figure 3 As shown, such QE variation 107 causes sub-pixel 18 to be substantially twice as sensitive to the incident NIR light band passed at filter passband 101 as sub-pixel 19 to be substantially more sensitive to the incident NIR light band passed at filter passband 102 .

[0038] The relative sensitivity of sub-pixels I8 and I9 can be further controlled by combining the respective bandpass filters 101, 102 with additional absorption filters (e.g., disposed within lens assembly 12 or between lens assembly 12 and sensor 14). The sensitivity of sub-pixel R can be controlled relative to the sensitivity of sub-pixels I8 and I9 by using an ND filter to match the range of the NIR pixel.

[0039] The lens assembly 12 is capable of focusing radiation towards the sensor 14 at all passbands 101, 102, 105. For example, the lenses of the assembly 12 may be apochromatic, tuned to the passbands 101, 102, 105. This will result in a properly focused image of the illuminated scene on the sensor plane for all wavelengths of interest.

[0040] Return to reference Figure 3, the NIR illumination source 16 includes at least a first LED or an array of first LEDs having an NIR emission band 103 centered at a wavelength of 875 nm and matching the passband 101 , and a second LED or an array of second LEDs having an NIR emission band 104 centered at a wavelength of 950 nm and matching the passband 102 .

[0041] In this embodiment, passbands 101 and 102 are narrower and sharper than corresponding LED emission bands 103 and 104. In this way, although LED emission bands 103 and 104 partially overlap at their edges, the NIR light emitted by the LEDs passing through the corresponding filters and directed to sub-pixel I8 is properly separated from the NIR light emitted by the LEDs passing through the filters and directed to sub-pixel I9. This reduces crosstalk between the sensor channels provided by sub-pixels I8 and I9.

[0042] However, in the context of the present disclosure, "matching" encompasses all cases where the LED emission band 103, 104 includes wavelengths that fall within the passband 101, 102, including cases where the emission band 103, 104 substantially coincides with or is narrower than the passband of the corresponding passband 101, 102.

[0043] The first LED and the second LED may be driven at different power levels by a dedicated circuit that may be controlled by the CPU 24 in such a way that the intensity of the emitted NIR light 103 is higher than the intensity of the emitted NIR light 104. In this way, the first LED and the second LED may be controlled separately with less leakage and less overlap between their emission bands 103, 104, thereby further improving crosstalk reduction between channels.

[0044] Visible light illumination source 17 comprises an LED or LED array having a bright red light emission band 106 centered at a wavelength of 700 nm and matching passband 105. Preferably, passband 105 is narrower and sharper than the corresponding LED emission band 106.

[0045] Return to reference Figure 1 In each acquisition cycle, image data acquired from the multi-band channels provided by sub-pixels I8, I9 and R can be written to the system memory 22 on the system bus 26 as needed by an application program executed by the CPU 24 or other dedicated processing blocks that can access the image sensor 14 and / or the memory 22.

[0046] In this embodiment, the system 10 also includes a dedicated face detector 18 for identifying facial regions from image planes acquired through the multi-band sensor channels. This functionality may also be implemented in software executed by the CPU 24. Data of the identified faces may also be stored in the system memory 22 and / or other memory, such as a secure memory or a database belonging to the system 10 or separate from the system.

[0047] In particular, system 10 is particularly adapted to operate face detection according to a first operating mode, wherein detector 18 is used to detect and track faces at different depths within an illumination scene using the NIR sensor channels provided by sub-pixel 18 and sub-pixel 19.

[0048] In practice, the lower sensitivity and number of sub-pixels I9 and the lower intensity level of emitted NIR light 104 cooperate to obtain a properly exposed image of faces near the system 10, such as the face of a vehicle driver and / or the face of a passenger next to the driver, while sub-pixel I8 will be substantially unaffected by the NIR light 104.

[0049] At the same time, the higher sensitivity and number of sub-pixels I8 and the higher intensity level of emitted NIR light 103 cooperate to obtain a properly exposed image of a face away from the system 10, such as the face of a vehicle occupant behind the driver, while sub-pixel I9 will be substantially unaffected by the NIR light 103.

[0050] Notably, having a greater number of sub-pixels 18 relative to the number of sub-pixels 19 improves the resolution of acquired images of distant faces, which are smaller and where the distance reflected NIR light travels to reach system 10 is greater than for close faces.

[0051] Ultimately, this combination of features enables both distant and close faces to be captured at the required level of detail as they can be accurately identified by the detector 18 .

[0052] For example, the detector 18 (or pre-detector) may implement a dedicated demosaicing (de-Bayering) module to reconstruct a complete image from the image plane acquired by the sensor IR channel. This module is aware of the different sensitivities of the different NIR bands 101, 102 associated with sub-pixels 18 and 19, and it may rely on a minimum maintained cross-correlation level between the channels. It should be understood that even the image component acquired in the NIR band 101 may be ignored when obtaining a demosaiced image of a close-up face. This functionality may also be implemented in software executed by the CPU 24 or other dedicated unit within the system 10.

[0053] Other image signal processing techniques may be customized for sensor 14 and applied to the acquired images, such as gamma correction and dynamic range compression to account for the characteristics of sensor 14 to appropriately render faces at different distances.

[0054] In this embodiment, the detector 18 is further configured to identify one or more eye regions and / or iris regions in the detected long-range and / or close-range faces. This functionality may also be implemented in software executed by the CPU 24. The data of the identified one or more iris regions may also be stored in the system memory 22 and / or other memory belonging to the system 10 or separate from the system.

[0055] In this way, the identified iris regions may be used as input to a biometric authentication unit (BAU) 20. Preferably, the BAU 20 is configured to extract an iris code from the received identified iris regions, and it may store the code in a memory 22 and / or other memory or database belonging to or separate from the system 10. In addition, the BAU 20 is preferably configured to compare the received one or more iris regions with reference iris regions associated with one or more predetermined objects (such as the owner of the vehicle and his or her family members), which reference iris regions may be stored in the memory 22, located in a secure memory within the BAU 20, or in any location accessible to the BAU 20.

[0056] An exemplary way of performing iris code extraction and comparison between iris regions is disclosed in WO2011 / 124512 (reference: FN-458) and involves using a master mask to compare between two image templates to select corresponding codes from the templates. The master mask excludes blocks and / or weights blocks from the matching process based on their known or expected reliability.

[0057] Other applications may benefit from the improved facial detection provided by the system 10 operating in the first operating mode, including fatigue, dullness and facial emotion detection, facial autofocus or defect detection and / or correction.

[0058] Figure 1 The system can also be operated according to a second operating mode for cardiac pulse detection. As is well known, the dilation and contraction of blood vessels corresponding to the heart rate causes periodic changes in the color of the illuminated skin. Pulse detection applications typically use only one visible color acquisition channel to monitor this periodic color change in the skin portion of the face being tracked.

[0059] An example of such an application is a webcam pulse detector that works with a PC webcam ( https: / / lifehacker.com / the-webcam-pulse-dector-shows-your-life-signs-using- y-1704207849). Pulse detection using visible light is generally unreliable, and detectability may vary with different skin colors. Furthermore, pulse detection reliability is low given that the detected color changes may be affected by surface skin color changes caused by factors unrelated to the pulse repetition frequency, such as ambient lighting and movement of the monitored person. For example, skin shading across the face is constantly changing, and if the person moves, these changes will exceed those due to the pulse repetition frequency, making detection unreliable. Euler video magnification may be applied to amplify periodic color changes visible in the face of consecutive frames of a video sequence ( http: / / people.csail.mit.edu / mrub / vidmag / ). Alternatively, neural network (NN) processing can improve the accuracy of camera-based pulse detection during natural (i.e., non-controller) human-machine interaction, such as disclosed in "A Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam" by Hamad Monkaresi et al. Here, visible light is used, and the acquired visual signal is not used directly for pulse detection, but is processed and subsequently subjected to independent component analysis.

[0060] In the second working mode of this embodiment, the detector 18 is used to detect and track the face of a person using the NIR sensor channel provided by the sub-pixel I8 and the sub-pixel I9 and the red light channel provided by the sub-pixel R as support.

[0061] The switch from the first operating mode to the second operating mode may occur periodically or may be triggered by a specific command / event.

[0062] When switching from the first operating mode to the second operating mode, the CPU 24 or another dedicated unit of the system 10 is capable of adjusting the image acquisition settings used in the first operating mode to acquire appropriately exposed images of the same face in different image planes provided by sub-pixel I8 and sub-pixel I9.

[0063] Specifically, when the face being tracked for pulse monitoring is in the vicinity of the system 10 (such as in the case where the face belongs to the driver of a vehicle or a passenger next to the driver), one or a combination of the following adjustments are performed:

[0064] - reducing the intensity of the NIR light 103 emitted by the LED,

[0065] - reduce the gain of sub-pixel I8;

[0066] - Reduce the accumulation (exposure) time.

[0067] It should be noted that using different accumulation times for pixels I8, I9 and R generally does not make it possible to use a rolling shutter exposure.

[0068] In any case, in this way, a properly exposed image of the tracked face can be formed in the image plane acquired from sub-pixel I8, while a properly exposed image of the same face can be formed in the image plane acquired from sub-pixel I9.

[0069] When the face being tracked for pulse monitoring is away from the system 10 (such as in the case where the face belongs to a vehicle occupant behind the driver), one or a combination of the following adjustments are performed:

[0070] - increasing the intensity of the NIR light 104 emitted by the LED,

[0071] - Increase the gain of sub-pixel I9;

[0072] - Increased build-up time.

[0073] In this way, a properly exposed image of the tracked face can be formed in the image plane acquired from sub-pixel I9, while a properly exposed image of the face can be formed in the image plane acquired from sub-pixel I8.

[0074] Additionally or alternatively, the relative exposure levels between the NIR sensor channels provided by sub-pixel 18 and sub-pixel 19 may be controlled by using additional filters that attenuate or amplify wavelengths of interest.

[0075] For the channel provided by sub-pixel R, CPU 24 may further adjust image acquisition settings to appropriately capture the tracked face, particularly in light of the distance of the face from system 10. Specifically, a lower intensity of LED-emitted red light 106 and / or a lower gain of sub-pixel R and / or a shorter accumulation time may be set for a closer face, while a higher intensity of LED-emitted red light 106 and / or a higher gain of sub-pixel R and / or a longer accumulation time may be set for a farther face.

[0076] When returning to the first operating mode, the CPU 24 or other dedicated unit can return to the image acquisition settings of the first operating mode.

[0077] Reference now Figure 1 The system 10 also includes a dedicated unit 30 for monitoring the heart pulse repetition frequency. This function can also be implemented in software executed by the CPU 24.

[0078] Specifically, unit 30 may access stored image data from detector 18 operating in the second operating mode, thereby making available for each of the sequence of image acquisition cycles:

[0079] - Image data of the monitored face from sub-pixel 18;

[0080] - Image data of the monitored face from sub-pixel 19; and

[0081] - Image data of the monitored face from sub-pixel R.

[0082] Thus, when all three sensor channels are available, unit 30 can track the evolution over time of the following three differential signals:

[0083] d1(t)=|V(t)-I8(t)|

[0084] d2(t)=|V(t)-I9(t)|

[0085] d3(t)=|I9(t)-I8(t)|

[0086] Wherein I8(t) is a time signal indicating the illumination of the monitored face obtained from sub-pixel I8 over time, I9(t) is a time signal indicating the illumination of the monitored face obtained from sub-pixel I9 over time, and V(t) indicates the illumination of the monitored face obtained from sub-pixel R over time.

[0087] Since the skin penetration depth depends on the wavelength of the incident radiation, the dilation and constriction of blood vessels caused by the heart rate will result in different illumination changes of the monitored face at the different frequency bands of the sensor channels provided by sub-pixels I8, I9 and R. On the other hand, changes in surface illumination due to other factors (e.g., changes in ambient lighting or changes in facial movement) tend to affect these multi-band channels in essentially the same way.

[0088] Thus, each of d1(t), d2(t) and d3(t) includes a non-zero component indicative of changes in illumination of the monitored face due to the pulse repetition frequency, while components within each of signals I8(t), I9(t) and V(t) due to other factors tend to cancel each other out in the calculated d1(t), d2(t) and d3(t).

[0089] In this way, the differential signals d1(t), d2(t) and d3(t) provide a more reliable measurement for monitoring the pulse repetition frequency, rather than just tracking illumination changes in one wavelength image acquisition channel. In addition, the pulse repetition frequency will be correlated with all the differential signals d1(t), d2(t) and 3(t), while the noise will be random. This aspect also means an increase in measurement accuracy.

[0090] Using NIR light also improves the measurement because it can penetrate the skin deeper than using visible light alone and thus allows better visualization of blood vessels. In addition, IR light is particularly suitable for monitoring the pulse repetition frequency of vehicle drivers because it can pass substantially through sunglasses compared to visible light.

[0091] The role of the signal R(t) is mainly supporting, especially considering the fact that the measurement conditions can change, resulting in over- or under-exposure of the images acquired through the sensor channels. In case the image from one of the channels is over- or under-exposed, the other two remaining channels can be used to properly perform pulse detection.

[0092] A frequency detection algorithm may be applied to the differential signals d1(t), d2(t), d3(t) to monitor the pulse repetition frequency, which may result in a determination of whether the pulse repetition frequency meets a critical threshold level or calculation of a pulse repetition frequency value. For example, autocorrelation and cross-correlation methods or other signal frequency detection methods, such as involving Fourier transforms, may be used.

[0093] Alternatively, artificial neural network processing may be used to perform heart pulse monitoring based on the differential signals d1(t), d2(t), d3(t).

[0094] In a variation of the above embodiment, instead of using a MSFA mounted on a CMOS sensor, the lens assembly 12 may be configured to filter and separate the incident radiation into spectral bands that are each focused on a corresponding different area of ​​a sub-pixel on the same sensor or sensor group, which may be used for multi-spectral image acquisition. An example of such an arrangement employing multiple lens barrels is disclosed in European Patent Application No. EP3066690 (Ref: IO006-0035-EP-01).

[0095] Alternatively, the MSFA filtering functionality may be achieved by configuring groups of sub-pixels of the image sensor itself, e.g., by appropriate choice of materials, to be selectively and differentially sensitive to correspondingly different frequency bands of incident radiation.

[0096] In other variations of the above embodiments, the sensor 14 may include more than two different sets of NIR-sensitive pixels to properly acquire faces at more than substantially two depth levels of the imaged scene, such as where a vehicle has more than two rows of occupants.

[0097] In other variations of the disclosed embodiments, the NIR illumination source 16 may include a single device capable of emitting different wavelengths, including at least an emission band 103 that matches the filter passband 101 associated with the subpixel 18 and an emission band 104 that matches the filter passband 102 associated with the subpixel 19, such as a laser or flashlight source. The relative intensities of such frequency bands within the emitted light may be controlled using a spectral filter or mask included in or disposed near the emission opening of the light source.

[0098] In other variations of the disclosed embodiments, subpixel R may be replaced by a subpixel sensitive to a different visible wavelength band (e.g., green light band) or by a subpixel sensitive only to white light. However, it should be understood that the presence of a subpixel for providing a visible light sensor channel is optional.

[0099] Although in the disclosed embodiment the system 10 is configured to switch between a first mode of operation and a second mode of operation, it will be appreciated that the functionality associated with these modes of operation may be implemented separately in a dedicated / separate image processing system.

[0100] In this case, the system dedicated to pulse repetition frequency monitoring may differ from the disclosed system 10 at least in that the number of sub-pixels I8 and I9 and / or the illumination intensity of the matching NIR light may be the same.

[0101] It should also be understood that the image processing functions of the disclosed system 10 can be implemented in the same processing unit or a group of processing units. In particular, in the case of vehicle DMS applications, such image processing functions can be usefully implemented in a multi-processor engine disclosed in U.S. Provisional Patent Application No. 62 / 592,665 (Reference: FN-618).

Claims

1. An image processing system (10), comprising: At least one image sensor (14), the at least one image sensor comprising a plurality of pixels, each pixel (100) comprising a plurality of sub-pixels (R, I8, I9) and configured to provide a first image plane and a second image plane during an image acquisition cycle, the first image plane being from a first sub-pixel group selectively sensitive to a first NIR light band (103), the second image plane being from a second sub-pixel group selectively sensitive to a second NIR light band (104), wherein the sensitivity of the first sub-pixel to the first NIR light is greater than the sensitivity of the second sub-pixel to the second NIR light band, and the number of the first sub-pixels is greater than the number of the second sub-pixels; at least one NIR light source (16), the at least one NIR light source being capable of emitting first NIR light corresponding to the first NIR light band and second NIR light corresponding to the second NIR light band, respectively, the first NIR light having a higher intensity than the second NIR light; as well as A face detector (18) is configured to detect at least a first face from the first image plane and a second face from the second image plane, respectively, wherein the first face is a face of a first person and the second face is a face of a second person different from the first person.

2. The image processing system according to claim 1, comprising: a first spectral bandpass filter (101) configured to pass light within the first NIR light band toward the first sub-pixel; and a second spectral bandpass filter (102) configured to pass light within the second NIR light band toward the second sub-pixel.

3. The image processing system of claim 2, wherein the image sensor comprises a multi-spectral filter array, the multi-spectral filter array comprising the first spectral bandpass filter and the second spectral bandpass filter. 4 . The image processing system of claim 1 , wherein the first NIR light band and the second NIR light band are centered at a first wavelength and a second wavelength, respectively, the first wavelength being lower than the second wavelength. 5 . The image processing system of claim 4 , wherein the first wavelength is 875 nm and the second wavelength is 950 nm. 6 . The image processing system of claim 1 , wherein the at least one light source comprises at least one first LED configured to generate the first NIR light and at least one second LED configured to generate the second NIR light. 7 . The image processing system of claim 1 , further comprising an iris detector configured to detect irises within the detected first and second faces.

8. The image processing system according to claim 7, further comprising a biometric authentication unit (20) configured to recognize one or more predetermined objects based on irises detected within the first face and the second face.

9. An image processing system according to claim 1, wherein the system is configured to operate in a first operating mode, wherein the face detector is configured to detect the first face and the second face at spatially non-overlapping positions within the first image plane and the second image plane, respectively.

10. An image processing system according to claim 9, wherein the system is configured to operate in a second operating mode, wherein the face detector is configured to detect the first face and the second face at spatially overlapping positions within the first image plane and the second image plane.

11. The image processing system according to claim 10, configured to adjust, when switching from the first working mode to the second working mode: at least one of the intensity of the first NIR light, the gain of the first sub-pixel, and the image acquisition period in a manner such that the first face is properly exposed in the first image plane; or At least one of the intensity of the second NIR light, the gain of the second sub-pixel, and the image acquisition period is in a manner such that the second face is properly exposed in the second image plane.

12. The image processing system according to claim 10 further includes a heart rate pulse monitoring unit (30), which is configured to analyze at least a first differential signal over the image acquisition cycle sequence, wherein the first differential signal indicates the difference in illumination between the detected first face and the second face.

13. The image processing system of claim 12, wherein the at least one image sensor is further configured to acquire a third image plane from a third sub-pixel group of the pixel array during each image acquisition cycle, the third sub-pixel being selectively sensitive to a visible light band, The system further comprises at least one visible light source (17) capable of emitting light corresponding to the visible light band, The face detector is configured to detect a face within the third image plane at a position coincident with the first face and the second face, and The heart rate pulse monitoring unit is further configured to analyze at least a second differential signal over the sequence of image acquisition cycles, the second differential signal indicating a difference in illumination between the detected third face and at least one of the detected first face and the second face.

14. The image processing system of claim 1, wherein the at least one image sensor is arranged to operate in a rolling mode shutter, wherein each of the image planes is acquired with a common accumulation time.

15. An image processing system (10), comprising: At least one image sensor (14), the at least one image sensor comprising a plurality of pixels (100), each pixel comprising a plurality of sub-pixels (R, I8, I9) and configured to provide a first image plane and a second image plane during an image acquisition cycle, the first image plane being from a first sub-pixel group selectively sensitive to a first NIR light band (103), the second image plane being from a second sub-pixel group selectively sensitive to a second NIR light band (104), wherein the sensitivity of the first sub-pixel to the first NIR light is greater than the sensitivity of the second sub-pixel to the second NIR light band, and the number of the first sub-pixels is greater than the number of the second sub-pixels; at least one NIR light source (16), the at least one NIR light source being capable of emitting first NIR light corresponding to the first NIR light band and second NIR light corresponding to the second NIR light band, respectively, the first NIR light having a higher intensity than the second NIR light; as well as Face detector (18); The system is configured to operate according to at least a first operating mode, wherein the face detector is configured to detect at least a first face in the first image plane and a second face in the second image plane at a location that does not spatially coincide with the first face, The first face is a face of a first person, and the second face is a face of a second person different from the first person.

16. An image processing system (10), comprising: At least one image sensor (14), the at least one image sensor comprising a plurality of pixels (100), each pixel comprising a plurality of sub-pixels (R, I8, I9) and configured to provide a first image plane and a second image plane during an image acquisition cycle, the first image plane being from a first sub-pixel group selectively sensitive to a first NIR light band (103) and the second image plane being from a second sub-pixel group selectively sensitive to a second NIR light band (104); at least one NIR light source (16), the at least one NIR light source being capable of emitting first NIR light corresponding to the first NIR light band and second NIR light corresponding to the second NIR light band, respectively; a face detector (18) configured to detect a first face from the first image plane and a second face from the second image plane at spatially coincident locations, wherein the first face is a face of a first person and the second face is a face of a second person different from the first person; and A heart rate pulse monitoring unit (30) is configured to analyze at least a differential signal over a sequence of image acquisition cycles, the differential signal being indicative of a difference in illumination between the detected first face and the second face.

17. One of a portable electronic device or a vehicle occupant monitoring system comprising the image processing system according to claim 1, 15 or 16.

Images