Driver monitoring system and method

Abstract

A system and corresponding method for monitoring biomedical parameters are described. The system includes a light source device providing a coherent illumination pattern that generates an illumination spot on a body surface of an individual, a camera unit that collects images of the coherent illumination spot, and a control unit including at least one processor. The control unit receives input data including color images and processes the input data to determine one or more biomedical parameters of the individual. The processing includes determining a temporal overlap between an acquisition time and a selected pulse sequence of illumination and selecting an image sequence having an overlap exceeding a selected threshold. The sequence of images is processed to identify pixels associated with the coherent illumination spot. A contrast variation function indicative of variations in speckle contrast is generated, and the one or more biomedical parameters of the individual are determined based on the contrast variation function.

Description

[Technical Field]

[0001] The present disclosure relates to systems and methods for monitoring biomedical parameters of individuals, and more particularly to monitoring one or more parameters of one or more individuals in a vehicle. [Background technology]

[0002] A driver monitoring system (DMS) is a vehicle safety system used to analyze and monitor the driver's condition and to warn or take action to reduce the risk of a vehicle collision. A typical DMS utilizes one or more camera units and a control system operable to collect and analyze data regarding the driver's attention, alertness, and possibly other biomedical conditions.

[0003] DMS systems are considered high-level safety systems for road vehicles. Such systems are among the list of safety systems required to provide reliable monitoring of driver attention and awareness. This is particularly important in vehicles that utilize assisted driving functions.

[0004] VCSEL technology is becoming an increasingly popular choice for implementation in automotive production due to its high power, robustness, and cost-effectiveness. Furthermore, the VCSEL's structure results in a narrow bandwidth compared to LEDs, providing improved temporal coherence and signal-to-noise ratios when needed. The use of a VCSEL light source also allows for selective filtering to avoid disturbances from ambient lighting and focus the VCSEL's narrow emission bandwidth. In recent years, driver monitoring systems (DMS) have been developed to provide real-time assessment of driver presence and status. These DMS include camera-based systems aimed at the driver's face to provide warnings to the driver and initiate intervention to manage vehicle control when necessary. This technology helps improve safety on the roads and provides better control over the vehicle.

[0005] A speckle pattern is a pattern of bright and dark areas formed within an illuminated spot due to optical coherence. Although speckle is typically considered noise, various techniques are known that utilize speckle analysis for remote monitoring of selected parameters. Laser speckle contrast analysis (LASCA) is a technique that allows for the monitoring and analysis of blood perfusion and blood flow in tissue based on variations in the contrast of the speckle pattern. This technique allows for the determination of data such as heart rate, heart rate variability, blood perfusion, and possibly blood pressure and respiratory rate. Summary of the Invention

[0006] Driver monitoring systems are becoming an increasingly important safety element in vehicles. Typical driver monitoring systems often utilize image processing to determine the driver's state and alertness level to improve road safety.

[0007] Monitoring a driver's biomedical and physiological parameters can provide valuable data about the driver's condition. Speckle analysis, particularly laser speckle contrast analysis (LASCA) technology, allows for the remote and non-contact collection of biomedical and physiological parameters such as heart rate, heart rate variability (HRV), respiratory rate, and blood perfusion level.

[0008] Determining such parameters using spackle analysis generally requires coherent illumination, which results in spackle patterns due to self-interference between light components. The present disclosure provides a light source unit configured to provide an illumination pattern including coherent illumination while being adapted for use in a typical vehicle space and for integration with existing typical driver monitoring systems (DMS). It should be noted that the disclosed system and driver monitoring systems utilizing the disclosed technology may also be used in off-road environments. Such systems may be used in cars, tractors, trains, airplanes, boats, or any vehicle that travels on land, water, and air. Furthermore, the system may be installed in a helmet with necessary size and range modifications.

[0009] The present disclosure provides a system for monitoring one or more biomedical and / or physiological parameters of an individual. The system is generally configured to operate within a driver monitoring system configuration, providing combined functionality in detecting and assessing driver attention, position, gaze direction, etc., and providing appropriate warnings as needed.

[0010] The disclosed system provides for monitoring one or more biomedical parameters of one or more individuals, typically a driver and / or passengers. The system includes a light source device that emits an illumination pattern that includes one or more coherent illumination beams and typically also includes some incoherent illumination. The system further includes a camera unit configured to acquire color images of selected areas aligned with the area illuminated by the light source device. The system further includes a control unit that includes at least one processor and memory and carries computer-readable instructions that, when executed by the processor, cause the control unit to perform the tasks described herein. Generally, the control unit is adapted to operate the light source device to generate a selected pulse sequence to illuminate the selected area, operate the camera to collect image data from the selected area, and receive the image data pieces from the camera unit. The control unit is operative to analyze the image data pieces and determine images associated with selected lighting conditions suitable for analyzing driver perception and other images suitable for determining biomedical data of the driver and / or one or more passengers. Typically, the control unit can classify the image data pieces based on their temporal overlap with the selected illumination sequence. This allows the image data fragments classified for analyzing driver perception to maintain the desired image quality because an appropriate coherent illumination pattern is used for speckle analysis and detection of biomedical and / or physiological parameters based on the image data fragments classified for this purpose. Thus, the present disclosure provides enhanced functionality and can be implemented in a driver monitoring system at a relatively low cost.

[0011] Thus, according to a broad aspect, the present disclosure provides a system for monitoring biomedical parameters of an individual, the system comprising: (a) a light source device that provides at least a coherent illumination pattern to generate one or more coherent illumination spots on a body surface of the individual; (b) a camera unit operable to collect one or more color images, the camera unit having a position for collecting image data from the scene, the one or more color images including at least a first subsequence of images including the one or more coherent illumination spots; (c) a control unit including at least one processor adapted to operate the light source device and the camera unit, to receive input data from the camera unit including one or more color images, and to process the input data to determine one or more biomedical parameters of the individual; The processing includes determining a temporal overlap between the acquisition time of color images in the input data and a selected pulse sequence of illumination emitted by the light source device and selecting a sequence of images having an overlap exceeding a selected threshold; processing the sequence of images to identify pixels associated with the one or more coherent illumination spots on the body surface of the individual; generating a contrast variation function indicative of variation in speckle contrast in the coherent illumination spots in the sequence of images; and determining one or more biomedical parameters of the individual based on the one or more contrast variation functions.

[0012] According to some embodiments, the light source device comprises an array of single-mode VCSEL units configured to emit a plurality of coherent illumination beams to generate the one or more coherent illumination spots on a body surface of the individual.

[0013] According to some embodiments, the light source device is configured to generate the one or more coherent illumination spots on the body surface of the individual at selected wavelengths between 800 nm and 1100 nm.

[0014] According to some embodiments, the light source device further comprises one or more arrays of VCSEL units, including multimode VCSEL units.

[0015] According to some embodiments, the light source device further comprises one or more LED light sources.

[0016] According to some embodiments, the camera unit comprises an RGB sensor.

[0017] According to some embodiments, the camera unit comprises an RGB-IR sensor carrying RGB pixels and infrared sensitive pixels.

[0018] According to some embodiments, the camera unit comprises a partial bandpass spectral filter configured to transmit near-infrared illumination in one or more selected wavelength ranges and reduce the intensity of visible range light to enhance detection of near-infrared illumination in the color image.

[0019] According to some embodiments, the camera unit is operable at a variable exposure rate to acquire image data, and the camera unit is operable with an image collection pattern including at least one sub-pattern synchronized with an illumination sequence of the one or more coherent illumination spots to collect the sequence of images.

[0020] According to some embodiments, the image collection pattern further comprises one or more sub-sequences of image acquisition that include images suitable for image processing.

[0021] According to some embodiments, the image processing includes one or more of head detection, eye tracking, body position detection, blink detection, and facial keypoint identification.

[0022] According to some embodiments, the system may be configured as a driver monitoring system (DMS), where the individual is the driver of the vehicle.

[0023] According to some embodiments, the system may be configured as a driver monitoring system (DMS) and / or an occupant monitoring system, where the individual is at least one occupant of the vehicle.

[0024] According to some embodiments, the light source system may further comprise one or more incoherent light sources operable for at least one of flood illumination and spatially separated pattern illumination.

[0025] According to some embodiments, the one or more color images include one or more second sub-sequences of color images associated with incoherent illumination.

[0026] According to some embodiments, the light source device comprises one or more VCSELs selectively operable to emit incoherent illumination in accordance with modulation of current input to the VCSELs.

[0027] According to one other broad aspect, the present disclosure provides a method, comprising: (a) providing an image data sequence of a region of interest comprising at least a portion of an individual's body, and providing data indicative of a sequence of coherent illumination pulses emitted by a light source device; (b) processing the image data sequence to determine a subsequence of images that overlaps in time with the sequence of coherent illumination pulses; (c) processing the sub-sequence of images to determine one or more illumination spots associated with coherent illumination within the sub-sequence of images; (d) utilizing the one or more illumination spots to determine a measure of contrast variation between images of the subsequence; determining a contrast variation function indicative of contrast variation of the one or more illumination spots in the subsequence of images; and determining one or more biomedical parameters of the individual based on the contrast variation function.

[0028] According to some embodiments, the method may further include determining a plurality of two or more contrast variation functions associated with two or more illumination spots in the subsequence of images, and for each of the contrast variation functions, determining a time-varying quality factor, and determining one or more biomedical parameters of the individual includes selecting a contrast variation function having a quality factor above a selected threshold within one or more respective time windows.

[0029] According to some embodiments, the data indicative of the sequence of coherent illumination pulses emitted by the light source device includes data regarding a sequence of operation of an array of single-mode VCSEL units configured to emit a plurality of coherent illumination beams to generate the one or more coherent illumination spots on a body surface of the individual.

[0030] According to some embodiments, the method may further include operating the light source device in a first mode to provide coherent illumination and in a second mode to provide incoherent illumination.

[0031] According to some embodiments, the method may further include operating the light source device in the second mode including providing a modulated input current to the light source device to improve the linewidth and reduce the coherence of the emitted light.

[0032] According to some embodiments, the coherent illumination pulses emitted by the light source device include illumination at a selected wavelength between 800 nm and 1100 nm.

[0033] According to some embodiments, the image data sequence includes color images.

[0034] According to some embodiments, the image data sequence includes images collected using an RGB-IR sensor carrying RGB pixels and infrared-sensitive pixels, and the image data sequence includes images collected through a partial bandpass spectral filter configured to transmit one or more selected near-infrared wavelength ranges and reduce the intensity of visible range light to enhance detection of near-infrared illumination in the color images.

[0035] According to some embodiments, the image data sequence includes images collected using a variable exposure rate to acquire the image data, and the subsequence of images is synchronized with the sequence of coherent illumination pulses emitted by the light source device.

[0036] According to some embodiments, the exposure time of the image associated with the sequence of coherent illumination pulses is in the range of 50 μs to 300 μs.

[0037] According to some embodiments, the image data sequence includes one or more second sub-sequences of images collected for use in image processing.

[0038] According to some embodiments, the exposure time of the sub-sequence of image data directed to the DMS is in the range of 0.5 ms to 5 ms.

[0039] According to some embodiments, the method may further include processing the one or more second sub-sequences of images for one or more of head detection, eye tracking, body position detection, blink detection, and facial keypoint identification. [Brief explanation of the drawings]

[0040] In order to better understand the subject matter disclosed herein, and to illustrate how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Figure 1]1 illustrates a system according to some embodiments of the present disclosure. [Figure 2A] 2A and 2B illustrate light source devices according to some embodiments of the present disclosure, including a single-mode VCSEL array (FIG. 2A) and a composite VCSEL array including single-mode and multimode sub-arrays (FIG. 2B). [Figure 2B] 2A and 2B illustrate light source devices according to some embodiments of the present disclosure, including a single-mode VCSEL array (FIG. 2A) and a composite VCSEL array including single-mode and multimode sub-arrays (FIG. 2B). [Figure 3] 1 illustrates an example of a temporal illumination pattern, according to some embodiments of the present disclosure. [Figure 4] 1 illustrates classification of image data based on illumination patterns, according to some embodiments of the present disclosure. [Figure 5] 1 illustrates machine learning-based processing of multiple contrast variation functions according to some embodiments of the present disclosure. [Figure 6A] 6A illustrates the spectral response function of an RGB camera, where FIG. 6A shows the spectral response function and typical illumination wavelengths, and FIG. 6B also shows a bandpass filter according to some embodiments of the present disclosure. [Figure 6B] 6A illustrates the spectral response function of an RGB camera, where FIG. 6A shows the spectral response function and typical illumination wavelengths, and FIG. 6B also shows a bandpass filter according to some embodiments of the present disclosure. [Figure 7] 1 illustrates a modified Bayer filter according to some embodiments of the present disclosure. [Figure 8] 1 illustrates in block diagram form a method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0041] As described above, the present disclosure provides systems and methods usable for detecting biological, biomedical, and / or physiological parameters of an individual. The present technology provides for monitoring an individual's vital signs, including heart rate (HR), heart beat interval (IBI), heart rate variability (HRV), respiratory waveform, respiratory rate (RR), and blood perfusion (BP), and is generally configured to operate in a vehicle as part of a vehicle management system, including for on-road and / or off-road use. More specifically, the present disclosure provides a driver monitoring system with known functionality, such as driver recognition and gaze detection data, that enables additional functionality that can be used to detect driver health conditions, such as stress, fatigue, and driver thermal comfort, with no or minimal loss of data quality.

[0042] In this regard, reference is made to FIG. 1 , which schematically illustrates a system according to some embodiments of the present invention. The system 50 may be configured to be mounted in a vehicle to monitor the driver's awareness and selected biomedical and / or physiological parameters of the driver. The system 50 includes at least one light source 100, at least one camera 200, and a control unit 300. Generally, the at least one light source 100 is operable to generate a selected illumination pattern OL including one or more spatial and temporal patterns directed toward an area where the driver 10 and, optionally, one or more passengers are located. The at least one camera unit is configured and operable to collect image data pieces including the illuminated area at a selected imaging rate and with a selected (optionally variable) exposure time. The camera unit 200 thus collects image data pieces including the reflected illumination pattern IL. The control unit 300, which typically includes at least one processor and memory and carries selected computer-readable instructions, is configured to receive the image data pieces from the camera 200 and process the image data pieces to determine one or more parameters of the at least one individual 10. The control unit 300 is thus operable to process image data fragments comprising one or more sequences of image data fragments in order to remotely monitor particular characteristics of the individual 10 .

[0043] In this regard, it should be noted that certain image processing and / or cutting operations may be performed within camera 200 itself. For example, in some embodiments, camera 200 may operate in a cropped mode to transmit image portions associated with one or more selected regions of the image, including associated data regarding features such as speckle patterns, the driver's facial region, the driver's forehead, cheeks, or other body parts. The use of cropped image data may provide for saving data transmission bandwidth within the system.

[0044] Such image cropping functions may be performed on different image fragments according to an image sequence and / or an illumination sequence. For example, an image sequence may include image data fragments collected for speckle analysis and image data fragments collected for general DMS analysis. For such DMS image data fragments, the camera 200 may operate to crop selected image portions for image processing and determine one or more frame regions where speckle pattern analysis may provide enhanced data, such as the foreheads of one or more persons in a vehicle. The camera may utilize such data to improve image cropping for use in speckle analysis and simplify the amount of image processing and data transmission. Also, as illustrated herein, images intended for spackle analysis are typically collected using coherent illumination, while images intended for DMS analysis are typically collected using ambient or incoherent illumination.

[0045] In this regard, the system 50 of the present disclosure is preferably configured to provide DMS functionality and may be provided as an add-on including hardware and / or software functionality to a DMS. For example, the light source device 100 is generally a lighting unit that directs coherent illumination onto specific regions of the subject's body to generate one or more illumination spots. Additionally, the light source device 100 may provide flood lighting and / or additional lighting patterns that provide desired imaging conditions that enable the control unit to determine driver parameters.

[0046] Additionally, camera 200 is typically a color camera, including, for example, an RGB sensor or a Bayer filter. Camera 200 is also preferably configured to collect near-infrared data, allowing for detection of additional parameters and enabling the use of infrared illumination patterns to protect the driver's eyes and avoid distractions while driving. Camera 200 is typically configured to operate at a selected frame rate (e.g., 60-150 fps) and can utilize constant and / or variable exposure times for different pieces of image data.

[0047] In general, image data pieces (frames) may be divided into a selected ratio between a first subsequence associated with coherent illumination and biomedical parameter detection and one or more second subsequences for DMS use. For example, camera 200 may be capable of operating at a frame rate of 60-150 fps, with frames split equally between channels to support 30-75 fps for each subsequence, or may be split unequally, e.g., 10 fps for video, 50 fps for vitals, or any other ratio. The selection of frame rate division may be determined according to system load and further requirements. More specifically, during stressful driving, the system may determine that the driver's attention to driving is relatively low and increase the DMS frame rate. Alternatively, if the driver's biomedical parameters are near or exceed a selected vital sign threshold, additional frames are directed to this processing. Note that the selection of frame rate division also affects the lighting pattern, which may vary accordingly.

[0048] The control unit 300 operates the light source device 100 and the camera unit 200 as described above. The control unit is further operative to receive input image data from the camera unit and process the input image data using two or more processing paths. More specifically, the control unit may determine, for each image slice, data regarding an illumination pattern associated with the image, with images collected with flood illumination being used for image processing such as gaze detection and other DMS processing tasks. Images collected with illumination patterns for depth detection are used for three-dimensional detection processing, and images collected with coherent illumination are used for speckle detection and analysis to determine one or more biomedical and / or physiological parameters of a person within the vehicle.

[0049] In general, the control unit 300 may include at least one processor and memory unit, and respective input / output modules implemented in hardware and / or software protocols. The control unit 300 stores computer-readable instructions that, when executed by the processor, cause the control unit to operate and generate operational commands for the light source device 100 and / or camera 200 as described herein. Such operations may be implemented as hardware and / or software models and are described herein as being contained within the control unit.

[0050] Furthermore, the control unit may typically be operable for various DMS functions, including monitoring the driver's attention using eye or gaze detection, head orientation, blink rate, etc., and may be configured to generate an alert if the driver is detected to be reducing their driving attention below a selected threshold. Additionally, as described herein, the control unit may further include computer-readable instructions for determining and monitoring one or more biomedical and / or physiological parameters of the driver and / or other occupants within the vehicle.

[0051] In this regard, reference is made to FIGS. 2A and 2B, which illustrate light source devices according to some embodiments of the present disclosure. FIG. 2A schematically illustrates a light source unit 100 configured to provide a coherent illumination pattern according to some embodiments of the present disclosure. The light source unit 100 includes a substrate 110, typically fabricated as a multi-layer substrate having a layered arrangement forming a vertical cavity. The substrate carries a plurality of single-mode VCSELs 130 operable to emit optical radiation in a selected wavelength range. The light source unit 100 also includes an optical element 120, such as a lens array, a diffractive optical element, or the like, configured to steer the output beams 132 of the single-mode VCSELs to provide an arrangement of substantially collimated beams propagating in a non-parallel relationship toward a selected region, thereby generating a selected illumination pattern in the selected region.

[0052] FIG. 2B illustrates a composite light source unit 100 according to some further embodiments of the present disclosure. In this example, a common chip 110 carries an array 130 of single-mode VCSELs and an additional array 150 of multimode VCSELs. The single-mode VCSELs are associated with optical element 120, as described above, to provide an array of non-parallel, nearly collimated output beams 132. The array of multimode VCSELs may also be associated with optical element 124 and configured to generate output light in the form of multiple output beams 152. The illumination pattern of multimode VCSEL array 150 may be selected to provide spatially separated light spots to support three-dimensional analysis of the illuminated area. Alternatively, as shown in FIG. 2B, the illumination pattern may include overlapping output beams that generate flood illumination suitable for general imaging operations performed by a typical DMS system.

[0053] Typically, in some embodiments, the light source unit 100 and / or the multimode VCSEL array 150 may be operable to eliminate or at least significantly reduce speckle contrast while providing flood illumination. To eliminate or at least reduce speckle contrast, the coherence characteristics of the light components emitted by the different VCSELs must be moderated to reduce coherence between the different VCSELs. The coherence of the VCSELs can be intentionally reduced by providing high-frequency modulation of the operating input current to the VCSEL. For example, the current modulation may be at a rate of 50 kHz or greater. The current modulation may be below and / or above the lasing threshold, or may be both above and below the lasing threshold. This modulation of the input current increases the laser linewidth to the 3-4 nm range, resulting in a corresponding reduction in coherence. In some embodiments, input current modulation may also be used on the VCSEL array 130 during periods intended for flood illumination, such as when operating a single VCSEL array 130 as illustrated in FIG. 2A. Current modulation may be used at selected times, for example, as illustrated in Figure 3, which shows a periodic modulation between coherent and incoherent illumination times. The use of modulation of the input current to the VCSEL arrays 130 and / or 150 may provide for dynamic operation of the light source device 100 based on selected and / or specific requirements, thereby improving the functionality and effectiveness of the driver monitoring system.

[0054] Furthermore, in some additional embodiments, the light source device may also include one or more LED light sources of selected wavelength ranges, which may be used in addition to an array of single-mode VCSELs as shown in Figure 2A and / or in addition to a combined single-mode and multimode VCSEL array as shown in Figure 2B.

[0055] An example of a temporal operating pattern for a light source device is shown in FIG. 3. As shown, the illumination pattern can include a series of short periods (e.g., 50-300 μs) of coherent illumination (CIL) and a series of longer periods (e.g., 3-30 ms) of general incoherent illumination (NIL). In some embodiments, the illumination sequence can include a sequence of incoherent flood illumination suitable for general image processing, gaze detection, etc., and a sequence of illumination patterns formed from spatially separated illumination spots suitable for three-dimensional mapping of the illuminated area. Note that flood illumination can be used at night or in low-light conditions, but may not be needed in daylight conditions and therefore can be turned off. In general, as described above, the incoherent illumination NIL periods can be achieved using modulation of the input current to the VCSEL array, which is used to increase the VCSEL linewidth and thereby reduce the coherence of the light emitted by the VCSELs in the array.

[0056] In general, the control unit 300 can operate the light source device with a selected lighting sequence. Furthermore, the control unit 300 can operate the camera 200 with a selected frame rate and a selected exposure time for each frame. This allows the control unit 300 to classify pieces of image data as associated with particular lighting conditions. FIG. 4 illustrates classification of an image sequence according to some embodiments of the present disclosure. As shown, an image sequence 401 is collected by the camera 200 and transmitted to the control unit 300. The control unit operates to classify pieces of image data as associated with a coherent lighting sequence CIL based on temporal overlap with the lighting sequence, or as associated with “normal” lighting NIL, which may include, for example, no lighting, flood lighting, and / or other lighting patterns. The control unit 300 operates to process the “normal” lighting images according to various DMS processing requirements, such as gaze detection, driver recognition, and face recognition. Furthermore, the control unit 300 is operative to process the coherently illuminated image sequences using spackle pattern analysis for detection of biomedical and / or physiological parameters such as heart rate, heart rate variability, respiratory rate, blood perfusion, etc.

[0057] It should be noted that the number of frames associated with different sub-sequences may be dynamically determined based on various requirements such as road safety and driver attention.

[0058] In this regard, the control unit 300 may operate to process the CIL image data fragments using speckle contrast analysis (LASCA). More specifically, the control unit may operate to detect one or more coherent illumination spots within the image data and determine the contrast of the speckle pattern at the identified illumination spots. The control unit 300 or its processor may then operate to determine, for the identified coherent illumination spots, a contrast variation function related to the variation in the contrast of the speckle pattern between image data fragments collected at different times. Variations in the contrast of the speckle pattern may indicate small vibrations or movements of the surface on which the speckle pattern is detected. Thus, for speckle patterns detected on the body surface of an individual, such as a driver, contrast variations may indicate capillary blood perfusion and other physiological variations at the body surface. Using known processing techniques, such as Fourier transform and frequency analysis, the control unit 300 may determine biomedical parameters of the driver.

[0059] In general, the processor may utilize various image processing techniques typically associated with DMS functions, such as face recognition, gaze detection head and / or eye tracking, key points on the face, body posture, etc., to enhance its ability to identify speckle patterns and their contrast. Furthermore, because the image data used for speckle analysis is based on color (RGB) image data, such processing functions can be easily implemented based on the control unit architecture and algorithms.

[0060] As shown in FIGS. 2A and 2B , the light source device can preferably provide an illumination pattern formed from multiple coherently illuminated spots. This provides the control unit 300 with multiple speckle patterns and determined biomedical parameters suitable for spackle pattern analysis. Accordingly, the processor can be operative to determine multiple contrast variation functions associated with each of the multiple illumination spots. To improve the quality of the biomedical data determined based on the speckle contrast analysis, the control unit can also be operative to determine a quality factor for each of the contrast variation functions. Such quality factors may vary over time, such as when the driver moves, their head turns, or the lighting changes. Therefore, the quality factor can be determined as a time-varying quality factor. Using the quality evaluation of the different contrast variation functions according to each function's time-varying quality factor, the control unit can determine an aggregated spackle contrast function and determine the driver's or passenger's biomedical parameters based thereon.

[0061] For example, the aggregated spackle contrast function may be determined based on one or more contrast variation functions having quality factors above a currently selected threshold. Additionally or alternatively, the control unit may utilize one or more machine learning (ML) or artificial intelligence (AI) processes to collect multiple contrast variation function signals and extract a pure physiological signal. Such ML / AI techniques may utilize one or more regression models that receive signals from N facial regions over a time period T.

[0062] The general operation of the ML / AI processing is illustrated in Figure 5. The processor first identifies a plurality of signals 511-51n exhibiting speckle patterns associated with respective plurality of illumination spots, and for each signal, the processor determines a varying contrast 521-52n and a contrast function 531-53n. The plurality of varying contrast functions 531-53n are provided as inputs to a regression model 540 to determine a selected vector 550 of biomedical parameters based on the speckle contrast analysis and the weight vector for each source.

[0063] The regression model may be trained using supervised or unsupervised training. In some preferred embodiments, the regression model may undergo unsupervised training, which causes the model to represent the signal and a clean spectral estimate of the signal that are close to each other in the embedding space. In this way, the model may use the coherent spectral features of the signal, which characterize the physiological component of the signal more than its noise component.

[0064] Further, as noted above, the system 50 of the present disclosure utilizes a color (e.g., RGB) camera 200. Furthermore, as noted above, the light source device 100 can provide general illumination using one or more LEDs and / or a multimode VCSEL array 150 for general illumination purposes and an array of single-mode VCSELs 130 for providing a coherent illumination pattern that produces a spackle pattern for biomedical parameter detection. The single-mode VCSEL array 130 is generally operable to emit infrared illumination at one or more selected wavelengths between 800 nm and 1100 nm to provide appropriate illumination that does not distract the driver. Figures 6A and 6B illustrate the spectral response of a typical RGB camera, illustrating optional infrared illumination wavelengths of 850 nm and 940 nm on the spectral response curve.

[0065] As illustrated in FIG. 6A, the RGB Bayer filters include pixels with different spectral responses, with transmission maxima around 450 nm, 540 nm, and 600 nm, respectively. Furthermore, as shown, the different filters have roughly similar transmittances for infrared wavelengths from approximately 820 nm to 1100 nm. Therefore, when operating at selected wavelengths between 800 nm and 1100 nm, the collection efficiency of the RGB camera is generally similar between pixels. To overcome the low response of infrared wavelengths relative to ambient visible light, the camera can operate with a shortened exposure time to collect image data associated with coherent illumination. For example, the exposure time for images aligned with coherent illumination can be as long as 50 to 300 μs, depending on the illumination pulse length. More specifically, returning to FIG. 3, the camera 200 can operate with an exposure time corresponding to a short illumination pulse for coherent illumination (CIL) and a longer exposure time for longer general illumination (NIL).

[0066] In some further embodiments, the camera 200 may utilize a bandpass filter SF, as shown in FIG. 6B. A bandpass filter may include one or more transmission bands and include lower collection intensities for other wavelength ranges. For example, as shown in FIG. 6B, a bandpass filter may allow transmission around selected wavelengths, such as 850 nm and 940 nm, and reduce transmission for visible wavelengths below 800 nm. The bandpass filter reduces the camera's sensitivity to visible light and increases its sensitivity to infrared illumination, improving detection of speckle patterns in the collected image data fragments.

[0067] An alternative configuration of camera 200 is illustrated in FIG. 7, which shows a modified Bayer filter MBF. The modified Bayer filter includes RGB pixels, while half of the green pixels are replaced by an infrared-transmitting filter, providing image pixels specifically selected for imaging in infrared illumination. Using a modified Bayer filter MBF as illustrated in this figure, the system of the present disclosure can obtain data for each pixel for processing in spackle analysis as described above, as well as RGB pixel data for other image processing associated with DMS functions.

[0068] As described above, the techniques of the present disclosure may be implemented by operation of a control unit including one or more processors and memory and carrying computer-readable instructions as described herein. In this regard, reference is made to FIG. 8 , which illustrates the main operational actions of the present disclosure. As shown, the method includes operating a light source device (8010) to generate an illumination pattern. The illumination pattern may include at least one coherent illumination sequence (8015) and may also include one or more incoherent illumination sequences, such as flood illumination, to provide appropriate imaging conditions and / or the selected illumination pattern to support three-dimensional mapping. In parallel with the illumination pattern, the method includes operating a camera (8020) to collect image data sequences. Preferably, the operation of the camera in image acquisition may be synchronized with the illumination sequence (8025). For example, as described above, the camera's exposure time may be aligned with the illumination period of the selected illumination condition, particularly the coherent illumination condition. For example, the exposure time of an image aligned with coherent illumination may be 50-300 μs long, depending on the illumination pulse length.

[0069] The method further includes processing the image data sequence (8030). The processing includes determining one or more subsequences of image data based on lighting conditions (8040). Generally, a first subsequence is associated with pieces of image data collected during a coherent illumination sequence. Additionally, the image data sequence may include one or more second subsequences associated with flood lighting (or natural lighting) and / or spatially separated illumination spot patterns.

[0070] The first subsequence is processed for detection of illumination spots having speckle patterns therein (8050) and for determining speckle contrast variation over time throughout the first subsequence (8060). The method may utilize ML and / or AI processing to determine a common signal sequence indicative of a person's biomedical parameters (e.g., driver and / or passenger) and / or for determining a quality factor of a contrast variation function between the speckle patterns determined at different illumination spots. Using the various contrast variation functions, the method operates to determine one or more biomedical parameters of one or more individuals within the vehicle (8070). Such parameters may include heart rate, heart rate variability, respiratory rate, blood perfusion, etc.

[0071] Operating in parallel with the processing of the first subsequence of image data, the method of the present disclosure may also operate to process one or more of the second subsequences in accordance with a selected DMS process 8080. The DMS process may be a typical DMS process that operates to detect a driver's attention and concentration level and may generate a warning, if necessary.

[0072] Biomedical parameter detection provides an additional level of functionality to the DMS, enhancing the system's functionality. Heart rate and heart rate variability detection allows the system to generate warnings if the driver is overly calm or overly stressed. The system can also be used to monitor additional occupants in the vehicle based on the alignment of lighting patterns with the camera's field of view.

[0073] Accordingly, the present disclosure provides systems, methods, computer-implemented methods, and / or program products operable to provide data regarding one or more biomedical and / or physiological parameters of one or more individuals. The systems of the present disclosure are generally adapted for use in vehicles and operate to monitor the driver's attention and awareness level, utilizing spackle pattern monitoring in addition to typical DMS processing.

Claims

1. 1. A system for monitoring biomedical parameters of an individual, the system comprising: (a) a light source device providing at least a coherent illumination pattern for generating one or more coherent illumination spots on a body surface of the individual; (b) a camera unit positioned to collect image data from a scene and operable to collect one or more color images, the one or more color images including at least a first subsequence of images including the one or more coherent illumination spots; (c) a control unit including at least one processor adapted to operate the light source device and the camera unit, to receive input data from the camera unit including one or more color images, and to process the input data to determine one or more biomedical parameters of the individual; The processing includes determining a temporal overlap between the acquisition time of color images in the input data and a selected pulse sequence of illumination emitted by the light source device, and selecting a sequence of images having an overlap exceeding a selected threshold; processing the sequence of images to identify pixels associated with the one or more coherent illumination spots on the body surface of the individual; generating a contrast variation function indicative of variation in speckle contrast in the coherent illumination spots in the sequence of images; and determining one or more biomedical parameters of the individual based on the one or more contrast variation functions.

2. 10. The system of claim 1, wherein the light source device comprises an array of single-mode VCSEL units configured to emit a plurality of coherent illumination beams to generate the one or more coherent illumination spots on a body surface of the individual.

3. The system of claim 1 or 2, wherein the light source device is configured to generate the one or more coherent illumination spots on the body surface of the individual at selected wavelengths between 800 nm and 1100 nm.

4. 4. The system of claim 2 or 3, wherein the light source device further comprises one or more arrays of VCSEL units, including multimode VCSEL units.

5. The system of any one of claims 2 to 4, wherein the light source device further comprises one or more LED light sources.

6. The system of any one of claims 1 to 5, wherein the camera unit comprises an RGB sensor.

7. The system of any one of claims 1 to 6, wherein the camera unit comprises an RGB-IR sensor carrying RGB pixels and infrared sensitive pixels.

8. 8. The system of claim 1, wherein the camera unit comprises a partial bandpass spectral filter configured to transmit near-infrared illumination in one or more selected wavelength ranges and reduce the intensity of visible range light to enhance detection of near-infrared illumination in the color image.

9. 9. The system of claim 1, wherein the camera unit is operable at a variable exposure rate to acquire image data, and wherein the camera unit is operable with an image collection pattern including at least one sub-pattern synchronized with an illumination sequence of the one or more coherent illumination spots to collect the sequence of images.

10. The system of claim 9 , wherein the image collection pattern further comprises one or more subsequences of image acquisition that include images suitable for image processing.

11. The system of claim 10 , wherein the image processing includes one or more of head detection, eye tracking, body position detection, blink detection, and facial keypoint identification.

12. A system according to any one of claims 1 to 11, configured as a Driver Monitoring System (DMS), wherein the individual is a driver of a vehicle.

13. The system of any one of claims 1 to 12, configured as a driver monitoring system (DMS) and / or an occupant monitoring system, wherein said individual is at least one occupant of a vehicle.

14. The system of any one of claims 1 to 13, wherein the light source system further comprises one or more incoherent light sources operable for at least one of flood illumination and spatially separated pattern illumination.

15. The system of any one of claims 1 to 14, wherein the one or more color images comprise one or more second sub-sequences of color images associated with incoherent illumination.

16. 16. The system of any one of claims 1 to 15, wherein the light source device comprises one or more VCSELs selectively operable to emit incoherent illumination in accordance with modulation by inputting current to the VCSELs.

17. 1. A method comprising: (a) providing an image data sequence of a region of interest comprising at least a portion of an individual's body, and providing data indicative of a sequence of coherent illumination pulses emitted by a light source device; (b) processing the image data sequence to determine a subsequence of images that overlap in time with the sequence of coherent illumination pulses; (c) processing the sub-sequence of images to determine one or more illumination spots associated with coherent illumination within the sub-sequence of images; (d) utilizing the one or more illumination spots to determine a measure of contrast variation between images of the subsequence; determining a contrast variation function indicative of the contrast variation of the one or more illumination spots within the subsequence of images; and determining one or more biomedical parameters of the individual based on the contrast variation function.

18. 20. The method of claim 17, further comprising determining a plurality of two or more contrast variation functions associated with two or more illumination spots in the subsequence of images, and for each of the contrast variation functions, determining a time-varying quality factor, and wherein determining one or more biomedical parameters of the individual comprises selecting a contrast variation function having a quality factor above a selected threshold within one or more respective time windows.

19. 19. The method of claim 17 or 18, wherein the data indicative of a sequence of coherent illumination pulses emitted by a light source device comprises data regarding a sequence of operation of an array of single-mode VCSEL units configured to emit a plurality of coherent illumination beams to generate the one or more coherent illumination spots on a body surface of the individual.

20. 20. The method of any one of claims 17 to 19, further comprising operating the light source device in a first mode to provide coherent illumination and in a second mode to provide incoherent illumination.

21. 21. The method of claim 20, wherein operating the light source device in the second mode comprises supplying a modulated input current to the light source device to improve the linewidth and reduce the coherence of the emitted light.

22. The method of any one of claims 17 to 21, wherein the coherent illumination pulses emitted by the light source device comprise illumination at a selected wavelength between 800 nm and 1100 nm.

23. The method of any one of claims 17 to 22, wherein the image data sequence comprises color images.

24. A method according to any one of claims 17 to 23, wherein the image data sequence comprises images collected using an RGB-IR sensor carrying RGB pixels and infrared sensitive pixels.

25. 25. The method of any one of claims 17 to 24, wherein the image data sequence comprises images collected through a partial bandpass spectral filter configured to transmit one or more selected near-IR wavelength ranges and reduce the intensity of visible range light to enhance detection of near-IR illumination in color images.

26. 26. The method of any one of claims 17 to 25, wherein the image data sequence comprises images collected using a variable exposure rate to acquire image data, and wherein the sub-sequence of images is synchronized with the sequence of coherent illumination pulses emitted by the light source device.

27. 27. The method of claim 26, wherein an exposure time of an image associated with the sequence of coherent illumination pulses is in the range of 50 μs to 300 μs.

28. 28. The method of claim 17 or 27, wherein the image data sequence comprises one or more second sub-sequences of images collected for use in image processing.

29. 29. The method of claim 28, wherein the exposure time of the sub-sequence of image data directed to the DMS is in the range of 0.5 ms to 5 ms.

30. 30. The method of claim 28 or 29, further comprising processing the one or more second sub-sequences of images for one or more of head detection, eye tracking, body position detection, blink detection, and facial keypoint identification.

Images