Infrared Light Source Protection System

Abstract

To avoid a risk of being exposed to the high-level infrared radiation for a long period of time.SOLUTION: Described herein is a monitoring system (100) including one or more infrared light sources (108, 110) for illuminating a subject (102) in a sequenced manner. The system (100) also includes a camera (106) for capturing images of the subject (102) during periods in which the subject (102) is illuminated by one of the light sources (108, 110). A processor processes the captured images to determine a brightness measure of the images and a controller controls the output power of the infrared light sources (108, 110) in response to the brightness measure. In response to the processor detecting a brightness measure below a predetermined brightness threshold, the controller is configured to switch off or reduce an output illumination intensity of one of the infrared light sources (108, 110).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present application relates to a control system, in particular to a control system for one or more infrared light sources.

[0002] Embodiments of the present invention are particularly adapted for controlling the power of infrared light sources in driver monitoring systems, however, it will be appreciated that the invention is applicable in wider contexts and other applications. [Background technology]

[0003] Humans are constantly exposed to varying levels of radiation across the electromagnetic spectrum, as shown in Figure 1.

[0004] Higher frequency radiation has more energy, can interact more strongly with materials it encounters, and is therefore generally more dangerous. For example, people can be constantly exposed to low frequency radio waves without any ill effects, but exposure to high frequency X-rays, even for a relatively short period of time, can be The range of frequencies generally considered to be dangerous to humans usually extends from ultraviolet to gamma radiation. However, there are some instances where lower frequency radiation, such as infrared radiation, may also be dangerous to humans.

[0005] Infrared light emission and sensing technology is used in many areas of technology today. Example applications include:

[0006] Treatment of sports injuries and burns using infrared light emitting diodes (LEDs).

[0007] Remote controls for TVs and other electronic devices.

[0008] Short-range communications such as between mobile phones and wireless headset systems.

[0009] A camera lighting device for focusing on a subject of interest.

[0010] Satellite imagery of clouds for weather forecasting.

[0011] Security and sensor systems.

[0012] Night vision device.

[0013] Face detection and recognition system. Summary of the Invention [Problem to be solved by the invention]

[0014] In many driver monitoring systems for vehicles, one or more infrared LEDs are used to emit infrared light into a scene that includes the driver's face. The reflected light is imaged by an infrared camera sensor into an image that is processed to sense the driver's drowsiness and / or attention level. The infrared radiation is invisible to the naked eye and therefore does not distract the driver while operating the vehicle. In such driver monitoring systems, the infrared LEDs are typically positioned approximately 30 cm to 1 m from the driver's face.

[0015] Unlike more powerful forms of electromagnetic energy in general, infrared radiation is typically It has only enough energy to initiate child movement, without breaking down molecules or causing tissue damage. When human tissue absorbs infrared radiation, the result is usually a feeling of warmth in the exposed area. Because infrared radiation acts to move molecules, moderate amounts of infrared radiation only heat nearby living tissue that is irradiated or touched.

[0016] However, in some cases, infrared radiation can be dangerous in that prolonged exposure to high levels of infrared radiation can cause burns, similar to exposure to a hot stove or another heat source, or prolonged exposure to the sun. The danger to the human body from excessive infrared radiation is caused by overheating of tissues, which can result in skin burns. Skin exposed to infrared radiation generally provides a warning mechanism for the effects of heat. A person may feel pain, but depending on the level of exposure to infrared radiation, the exposure may not cause immediate pain. For these reasons, standards such as IEC-62471 have been developed to regulate the operation of infrared emitting devices.

[0017] Protection against ultraviolet (and other harmful electromagnetic radiation) may be achieved by administratively regulated methods, such as limiting exposure time for employees in hazardous environments. Additionally, personal protective equipment, such as protective clothing, may be used. However, in applications such as driver monitoring, where continuous or near-continuous illumination of the driver with infrared radiation is advantageous, these methods may be impractical, and the inventors have recognized that other solutions must be found.

[0018] This is especially true when the infrared radiation source may be occluded by a part of the human body, such as the face or hand, which is positioned closely adjacent to the radiation source.

[0019] Any discussion of background art throughout this specification should not be taken as an admission that such art is widely known or forms part of the general knowledge in the field. [Means for solving the problem]

[0020] A preferred embodiment of the present invention is directed to offsetting the drawbacks of using infrared light sources in certain applications, such as driver or occupant monitoring systems that utilize face or eye detection / recognition / tracking systems, where an LED or other infrared light source is switched off or reduced in power when a human or other object is blocking the light source.

[0021] 1. A method for detecting an obstruction of at least one electromagnetic radiation source in a driver or passenger monitoring system including an imaging camera, comprising: (a) modulating the intensity of a source of electromagnetic radiation in a predetermined manner; (b) detecting whether said modulation is present in at least one region of a corresponding image of said imaging camera; (c) determining whether the radiation source is occluded based on the degree of modulation present.

[0022] In an embodiment, the modulating step (a) comprises a temporal or spatial modulation. In an embodiment, the image is divided into a number of tiled regions and the detecting step is applied to each tiled region.

[0023] In some embodiments, the modulation may include at least one of a pseudorandom modulation, a sinusoidal modulation, and a square wave modulation. In some embodiments, the detecting step may include applying a matching filter for the modulation to the captured image to determine if the modulation is present.

[0024] According to a further aspect of the present invention, one or more infrared light sources for continuously illuminating a subject; a camera that captures an image of the object while the object is illuminated by one of the light sources; a processor for processing the captured image to determine a brightness measurement of said image; a control device for controlling an output power of the infrared light source in response to the luminance measurement value; A monitoring system is provided in which the controller is configured to turn off or reduce an output illumination intensity of one of the infrared light sources in response to the processor detecting a degradation in performance below a predetermined brightness threshold.

[0025] In some embodiments the luminance measure is the average pixel intensity of the image. In some embodiments the luminance measure is the average pixel intensity of a subset of the pixels of the image. In some embodiments the luminance measure is the average pixel intensity of a region of pixels in the image. The pixel region preferably corresponds to the face, eye or eyes of the subject.

[0026] In some embodiments, the luminance measurement is a comparison of luminance between two or more images. In certain embodiments, the luminance measurement is a comparison of luminance between consecutive images. In other embodiments, the luminance measurement is a comparison of luminance between two images captured during illumination with a common infrared light source. In some embodiments, the comparison comprises comparing pixel intensities of the two or more images on a pixel-by-pixel basis. In other embodiments, the comparison comprises comparing average pixel intensities of the two or more images.

[0027] In some embodiments, when the infrared light source has been turned off for a predetermined delay period, the controller is configured to reactivate the infrared light source during a test period during which one or more test images are captured, and the controller is further configured to keep the infrared light source activated if a luminance measurement of the test image is greater than or equal to a predetermined luminance threshold, and to deactivate the infrared light source otherwise.

[0028] In some embodiments, at least one of the light sources is positioned at an angle greater than 3.2 degrees from the camera when viewed along the optical axis from the subject. Preferably, each of the light sources is positioned at an angle greater than 3.2 degrees from the camera when viewed along the optical axis from the subject.

[0029] In some embodiments, the output illumination intensity is determined by a look-up table stored in a database.

[0030] In some embodiments, in response to detecting a luminance measurement below a predetermined luminance threshold, the controller reduces the output illumination intensity and increases or decreases the illumination duration of the infrared light source based on a determination of radiation safety to personnel.

[0031] In some embodiments, the processor is configured to detect a current level of the luminance measurement from a plurality of predetermined levels, and in response, the controller is configured to set an output illumination intensity of one of the infrared light sources to a value corresponding to the current level.

[0032] In some embodiments, the system is mounted within the cabin of a vehicle and the subject is a driver or passenger of the vehicle.

[0033] According to a further aspect of the present invention there is provided a method of controlling a system of two or more infrared light sources, the method comprising: i. continuously illuminating a subject from one or more spaced infrared light sources; ii. Capture an image of the subject while it is illuminated by one of the infrared light sources. And, iii. processing the captured image to determine a luminance measurement of the image; iv. comparing the luminance measurement to a predetermined luminance threshold; v. controlling an output power of the infrared light source in response to the comparison of step iv, wherein in step v, in response to detecting a luminance measurement below the predetermined luminance threshold, The output illumination intensity of one of the light sources is reduced or set to zero. [Brief description of the drawings]

[0034] Exemplary embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings. [Figure 1] FIG. 1 is a diagram illustrating the electromagnetic spectrum and its primary subbands. [Diagram 2] FIG. 2 is a perspective view of the interior of a vehicle equipped with a driver monitoring system that includes a camera and two LED light sources. [Diagram 3] FIG. 3 is an explanatory diagram of the driver's viewpoint of the dashboard of a car equipped with the driver monitoring system of FIG. [Figure 4] FIG. 4 is a schematic functional diagram of the driver monitoring system of FIGS. [Diagram 5] FIG. 5 is a schematic plan view of the driver monitoring system of FIGS. 2-4 interacting with a driver, showing the illumination and imaging fields of view of the LEDs and camera along with each LED's respective attention zone. [Figure 6A] 6A and 6B are schematic plan views of the driver monitoring system of FIGS. 2 to 4 interacting with the driver at different distances. [Figure 6B] 6A and 6B are schematic plan views of the driver monitoring system of FIGS. 2 to 4 interacting with the driver at different distances. [Figure 7] FIG. 7 is a process flow diagram illustrating the major steps of a method for controlling a system of two or more infrared light sources. [Figure 8] FIG. 8 is a graph showing example LED pulsing curves for 5% and 10% duty cycles. [Figure 9] FIG. 9 shows an example of a case where only one LED is blocked. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] The protection system described herein may be applied and used in many environments. One example is monitoring the driver or passengers of an automobile or other vehicle, such as a bus, train, or airplane. Furthermore, the system described herein may be applied to an operator using or operating any other device, such as a machine or a flight simulator. For ease of understanding, embodiments of the invention are described herein within the context of a driver monitoring system for a vehicle. Furthermore, although the infrared light source is described as being an LED, it will be understood that the invention is also applicable to other types of infrared light sources, such as a vertical-cavity surface-emitting laser (VCSEL).

[0036] <<System Overview>> 2-4, a driver monitoring system 100 is illustrated that captures images of a vehicle driver 102 while operating a vehicle 104. The system 100 is further adapted to perform various image processing algorithms on the captured images, such as face detection, facial feature detection, facial recognition, facial feature recognition, facial tracking, or facial feature tracking, such as tracking a person's eyes. Examples of image processing procedures are described in U.S. Patent No. 7,043,056 to Edwards et al., entitled "Facial Image Processing System," and assigned to Seeing Machines LLC, the contents of which are incorporated herein by cross-reference.

[0037] As best shown in FIG. 3, the system 100 includes an imaging camera 106 disposed on or within an instrument display of a vehicle dashboard 107 and oriented to capture images of the driver's face in the infrared wavelength range for identifying, locating and tracking one or more human facial features.

[0038] The camera 106 may be a conventional CCD or CMOS based digital camera having a two-dimensional array of light-sensitive pixels and, optionally, the ability to determine range or depth (via one or more phase detection elements). The light-sensitive pixels are capable of sensing electromagnetic radiation in the infrared range. The camera 106 may also be a three-dimensional camera, such as a time-of-flight (TOF) camera or other scanning or range-based camera capable of imaging a scene in three dimensions. In other embodiments, the camera 106 may be replaced by a pair of similar cameras operating in a stereo configuration and tuned to extract depth. The camera 106 is preferably configured to image in the infrared wavelength range, although it will be appreciated that in alternative embodiments the camera 106 may image in the visible range.

[0039] Still referring to FIG. 3, in the first embodiment, the system 100 also includes a pair of infrared light sources in the form of light emitting diodes (LEDs) 108 and 110, which are horizontally symmetrically arranged on the vehicle dashboard 107 at respective positions proximate to the camera. The LEDs 108 and 110 are adapted to illuminate the driver 102 with infrared radiation during the time that the camera 106 is capturing images, so as to highlight the driver's face to obtain a high quality image of the driver's face or facial features. By operating the camera 106 and the LEDs 108 and 110 in the infrared range, visual obstructions to the driver's attention can be reduced. The LEDs 108, 110 may be operated continuously, intermittently, or cyclically, and may be alternately operated in a strobe manner, which provides the operational advantage of reducing glare present in the images. The operation of the camera 106 and the LEDs 108, 110 are controlled by an associated controller 112, which includes a computer processor or microprocessor and a memory for storing and buffering captured images from the camera 201. In other embodiments, different types of light sources may be used in place of the LEDs.

[0040] As best shown in FIG. 3, the camera 106 and the LEDs 108 and 110 may be manufactured or constructed as a single unit 111 having a common housing. The unit 111 is shown mounted on the vehicle dashboard 107 and may be installed during the manufacture of the vehicle or subsequently as a post-market product. In other embodiments, the driver monitoring system 100 may include one or more cameras and light sources mounted anywhere suitable for capturing images of the head or facial features of the driver, subject and / or passenger within the vehicle. As an example, the camera and LEDs may be located on the vehicle's steering column, rearview mirror, center console, or driver's side A-pillar. Also, in some embodiments, more than one light source may be employed in the system. In the illustrated embodiment, the first and second light sources each include a single LED. In other embodiments, each light source may include multiple individual LEDs.

[0041] In the first illustrated embodiment, the LEDs 108 and 110 are preferably spaced horizontally at a distance ranging from about 2 cm to about 10 cm, and are positioned from about 30 cm to about 80 cm from the driver's face. The spacing of the LEDs 108 and 110 is variable, provided that the LEDs are positioned sufficiently off-axis from the camera such that the red-eye effect is not present in the captured image. Typically, the red-eye effect can be avoided if the LEDs illuminate the driver at an angle greater than about 3 degrees from the optical axis of the camera.

[0042] Some embodiments of the present invention are particularly adapted to operate in "dark pupil" conditions. Such conditions require that the light source be positioned at more than a certain angle from the camera, as viewed from the driver along the optical axis. The bright pupil effect is influenced by many factors, including pupil dilation (size), gaze angle relative to the camera, and wavelength of light. For example, a good rule of thumb for a good package size / signal performance balance at 950 nm is a 3.2 degree angle for the bright pupil effect. Can be used to indicate hole status.

[0043] In dark pupil conditions, the red-eye effect is eliminated or substantially reduced, and specular reflections on glasses do not overlap sufficiently to degrade tracking quality. Driver monitoring systems operating in dark pupil conditions can provide enhanced performance in terms of higher eyelid and gaze availability and accuracy. In some embodiments, system 100 is configured to operate when only one or a subset of the LEDs are positioned in a dark pupil condition.

[0044] Turning now to FIG. 4, functional components of the system 100 are illustrated in a schematic diagram. The system controller 112 serves as the central processor of the system 100 and is configured to perform a number of functions as described below. The controller 112 is located in the dashboard 107 of the vehicle 104 and may be connected to or integrated with the vehicle's on-board computer. In another embodiment, the controller 112 may be located in a housing or module along with the camera 106 and LEDs 108 and 110. The housing or module may be sold commercially and installed in the vehicle dashboard and then adjusted for use in that vehicle. In further embodiments, such as a flight simulator, the controller 112 may be an external computer or unit, such as a personal computer.

[0045] The controller 112 may be implemented as part of any form of computer processing device or device that processes electronic data, e.g., from registers and / or memory, and converts the electronic data into other electronic data that may be stored, e.g., in registers and / or memory. As shown in Figure 4, the controller 112 includes a microprocessor 114 that executes code stored in memory 116, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and other equivalent memory or storage systems that will be readily apparent to one of ordinary skill in the art.

[0046] The microprocessor 114 of the controller 112 includes a vision processor 118 and a device controller 120. Both the vision processor 118 and the device controller 120 represent functional elements performed by the microprocessor 114. However, it will be appreciated that in alternative embodiments, the vision processor 118 and the device controller 120 may be implemented as separate hardware, such as a microprocessor in combination with custom or specialized circuitry.

[0047] Vision processor 118 is configured to process the captured images to perform driver monitoring, such as to determine a three-dimensional head pose and / or gaze position of driver 5 within the monitored environment. To accomplish this, vision processor 118 utilizes one or more gaze determination algorithms, which may include, for example, the methods described in U.S. Patent No. 7,043,056 to Edwards et al., entitled "Facial Image Processing System," and assigned to Seeing Machines LLC. Vision processor 118 may also perform a variety of other functions, including determining attributes of driver 5, such as eye closure, blink rate, etc., and tracking the driver's head movements to detect driver attention, drowsiness, or other issues that may prevent the driver from safely operating a vehicle.

[0048] The raw image data, gaze position data, and other data acquired by the vision processor 118 The data is stored in the memory 116 .

[0049] The device controller 120 is configured to control the camera 106 to selectively activate the LEDs 108 and 110 sequentially in synchronization with an exposure time of the camera 106. For example, the LED 108 may be controlled to activate during odd image frames and the LED 110 is controlled to activate during even image frames to perform a strobe sequence. Other lighting sequences may be performed by the device controller 120, such as, for example, L,L,R,R,L,L,R,R... or L,R,0,L,R,0,L,R,0... where "L" represents the left mounted LED 108, "R" represents the right mounted LED 110, and "0" represents an image frame captured while both LEDs are not activated. LEDs 108 and 110 are preferably electrically connected to device controller 120, but may also be wirelessly controlled by controller 120 via wireless communications such as Bluetooth® or WiFi® communication.

[0050] Thus, during operation of the vehicle 104, the device controller 120 activates the camera 106 to capture images of the driver's 102 face in a video sequence. The LEDs 108 and 110 are activated and deactivated in synchronization with successive image frames captured by the camera 106 to illuminate the driver during image capture. The device controller 120 and vision processor 118 work in conjunction to provide for the capture and processing of driver images to derive driver state information, such as drowsiness, attention, and gaze position, during normal operation of the vehicle 104.

[0051] Additional components of the system may also be included within a common housing of unit 111 or may be provided as separate components according to additional embodiments. In one embodiment, operation of controller 112 is performed by an on-board computer system connected to camera 106 and LEDs 108 and 112.

[0052] <<Infrared radiation protection>> The present embodiment relates to a light source control method for reducing exposure of a subject to infrared radiation from an infrared radiation source, which may be implemented by an imaging system, such as a driver monitoring system 100, in which a subject is illuminated by one or more infrared light sources for imaging by a camera.

[0053] 5, a plan view of the system 100 interacting with a driver 102 is illustrated. In a typical situation, a vehicle driver using the dashboard mounted system 100 is generally far enough away from the LEDs 108 and 110 (which are light sources such that there is no infrared hazard or danger to the driver 230). However, there may be safety concerns if any parts or portions of the driver 230 are located too close or within a short distance from the infrared LEDs 204, 206. In this case, sufficient power density or energy may be emitted by the infrared LEDs to heat or burn human tissue, which may be similar to intense exposure to the sun on a sunny day.

[0054] The area around or at a distance from the infrared LEDs where there may be safety concerns is referred to as the "attention zone" 501 and 503, as illustrated in Figure 5. The size or distance of the attention zone varies depending on several factors, including but not limited to the average or peak power level of each infrared LED, the frequency emitted by the LEDs, and whether there are surfaces or objects near the infrared LEDs that reflect infrared energy. The attention zone or distance is typically 10 cm or less from the infrared LEDs. However, for powerful infrared LEDs or For strong light sources, the distance may be in the range of 15 cm or even more.

[0055] In a first embodiment, the detection of an object within the attention zone is estimated based on a brightness evaluation of the captured image. As shown in Figures 6A and 6B, as the driver approaches the LEDs and moves off-axis, more light is scattered away from the field of view of the camera 106. This results in the driver becoming darker, which in turn results in a decrease in the overall brightness of the captured image. The absolute brightness level or change in brightness of the image is evaluated against a predefined brightness threshold, resulting in the output power of the LEDs being controlled accordingly.

[0056] 7, there is shown a method 700 of controlling a system of two or more infrared light sources, such as system 100. For simplicity, method 700 will be described with reference to the operation of system 100. However, it will be appreciated that method 700 is also applicable to other imaging systems in which a subject is illuminated with infrared radiation from two or more infrared light sources and imaged with an infrared sensitive camera.

[0057] Method 700 includes an initial step 701 of sequentially illuminating a subject (e.g., driver 102) from two or more spaced infrared light sources (e.g., LEDs 108 and 110). As discussed above, the LEDs may be programmed to illuminate in a number of different sequences. However, in method 700, at least a subset of the captured images must be captured while the subject is illuminated by a single LED.

[0058] At step 702, an image of a subject is captured using the camera 106. In the case of the driver monitoring system 100, the image relates to the driver's face, including the driver's eyes for monitoring gaze and drowsiness. In some embodiments, the captured image is stored in memory 16, optionally after being subject to pre-processing by the vision processor 118 or on-board hardware of the camera 106 itself.

[0059] At step 703, the captured image is processed by vision processor 118 to determine a measure of the brightness of the image. The measure of brightness may be one or more of a number of brightness-related calculations, as described below.

[0060] In some embodiments, the brightness measure is a measurement of the average pixel intensity of each image. For grayscale images, pixel intensities in a typical camera sensor will be 8-bit digital integers ranging from 0 to 255, with 0 representing black and 255 representing white. Thus, in these systems, the average brightness value will have a value within this range determined by the sum of each pixel's intensity divided by the number of pixels. However, it will be understood that other camera sensors may record pixel intensities at higher or lower bit sizes, and thus record a larger or smaller intensity range.

[0061] If the camera sensor saves the image as a color image, then separate red, green, and blue components are assigned to each pixel. In a color image system, pixel intensities may be calculated as a vector of three numerical values ​​(red, green, and blue). In some embodiments, the three different color components are stored as three separate grayscale images, and pixel intensities are calculated from the grayscale version of the color image.

[0062] The measurement of the average pixel intensity may be performed for all pixels in an image to calculate an overall average pixel intensity, or may be performed only for a subset of the pixels in each image. The latter operation may be useful when an image contains less useful content, such as a dark background behind a driver's face. For example, the subset of pixels selected to determine the average pixel intensity may fall within a particular pixel region in the image. The pixel region may be , may correspond to a detected feature in the image, such as the driver / subject's face, eye, or both eyes. To determine the pixel region, the feature must first be identified. Traditional feature recognition techniques such as pattern matching, edge and contrast detection may be performed. Machine learning algorithms may also be employed to recognize the feature. Once a feature is identified, a bounding region around the feature may be specified to define an associated pixel region for which an average pixel intensity is to be calculated.

[0063] In other embodiments, the luminance measurement may include a comparison of luminance between two or more images. The comparison may be between consecutive images in an image stream, or between a predefined sequence of images in an image sequence. If the images were captured under an alternating left-right illumination sequence of LEDs (L,R,L,R,L,R...), then consecutive images will be illuminated by alternating LEDs. Thus, a comparison of brightness levels between successive images will indicate which LED is occluded, producing a lower brightness image. Comparing the brightness between successive odd or even images using the same illumination sequence will represent two images captured during illumination with a common infrared LED. This comparison can also indicate occlusion of that LED if there is a significant drop in brightness between successive images illuminated by that common LED.

[0064] In some embodiments, the comparison of luminance includes comparing two or more images. By way of example, the comparison may compare the luminance of an image illuminated by one LED to two or more previous images illuminated by that same LED. Alternatively, the comparison may compare the luminance of an image illuminated by one LED to two or more previous images illuminated by another LED. Also, the comparison may compare the luminance of as many previous images as there are, regardless of the illuminating LED.

[0065] In embodiments in which the brightness measurement involves a comparison of brightness between two or more images, the comparison may be made by comparing the pixel intensities of corresponding image pixels on a pixel-by-pixel basis, or by comparing the average pixel intensities of the different images (either the entire image or a subset of pixels).

[0066] In step 704, the vision processor 118 performs a comparison of the luminance measurement against a predefined luminance threshold stored in the memory 116. The luminance threshold may represent the average pixel intensity that must be achieved for a particular image, or may represent the difference in luminance between multiple images (if the luminance measurement is based on a luminance comparison between multiple images). The particular luminance threshold may depend on the geometry of the imaging system, such as the distance between the LEDs, the subject, and the camera, and the LED specifications (e.g., maximum output illumination intensity and pulse duration). By way of example, the luminance threshold may be equal to a particular average pixel intensity (between 0 and 255), such as 128, 100, 64. Alternatively, the luminance threshold may be equal to a percentage of the average pixel intensity calculated over a number of previous images, such as 75%, 50%, 40%, 30%, 25%, 20%, or 10% of the previous average pixel intensity.

[0067] In step 705, the output power of the infrared LEDs is controlled in response to the comparison performed in step 704. In response to detecting a brightness measurement below a predetermined brightness threshold, in step 705A, the device controller 120 turns off or reduces the output illumination intensity of one of the infrared LEDs. In particular, the LED that is determined to cause a reduction in the brightness of the image is switched off or has its output illumination intensity reduced. This is determined on the basis that the reduction in brightness is due to the subject being too close to the LEDs, casting a shadow in the image, as illustrated in FIG. 6B. Alternatively, it may be due to the occlusion of the subject in the image by an object.

[0068] If, at step 705, the brightness measurement equals or exceeds the predefined brightness threshold, then, at step 705B, the device controller 120 maintains the output illumination intensity of the LED.

[0069] The system operation described above is essentially binary, with the LED control being either in a high or low power state (or switched off completely) based on the measured brightness of the captured image. In other embodiments, a more dynamic control of the LED is provided, where the output power of the LED is controlled within a range of multiple power levels by the device controller 120 in response to the measured brightness. In this regard, in some embodiments, step 704 includes a comparison of the brightness measurement with one or more threshold levels. In these embodiments, multiple levels corresponding to different brightness threshold levels are defined, and the vision processor 18 is configured to determine which of these levels (the "current level") matches the brightness measurement. In response, the device controller 120 is configured to send a control signal to the LED to drive the LED at one of multiple output illumination intensity levels corresponding to the current level. By way of example, the brightness measurement may be compared to four brightness threshold levels with corresponding output illumination intensity levels at which the LED is driven. For example,

[0070] [Table 1]

[0071] It will be understood that fewer or greater threshold levels and corresponding output illumination intensity values ​​may be defined. The number of range bins of threshold brightness levels used and the appropriate LED output illumination intensity for each range bin may be determined by controller 112 and programmed by a user of the system. In some embodiments, the output illumination intensity as described above may be determined by a look-up table stored in memory 116.

[0072] As the illumination intensity increases or decreases, the LED's pulse time duration or illumination period can be relatively decreased or increased to keep the LED within the specified safe and stable operating limits. This determination of output illumination intensity and pulse duration is typically made based on a predefined pulse operation curve, such as that illustrated in FIG. 8. Such curves are typically based on determining stable operating conditions to avoid radiation safety and / or excessive heat damage to humans. An exemplary standard established for safe LED operation is IEC 62471.

[0073] A multiple threshold level system is provided in which the imaging system determines a threshold level based on brightness measurements of the captured image. It allows you to control the light source illumination intensity. It is a pseudo-measurement of the distance between the subject and the LED, based on the fact that objects closer to the LED's path cast darker shadows.

[0074] During periods when one of the LEDs is not activated, device controller 120 may be configured to capture images at a lower frame rate such that images are captured only during illumination by the remaining activated LED or LEDs, or vision processor 120 may be configured to ignore images captured during periods when the inactivated LEDs were previously controlled to illuminate the image.

[0075] The system 100 is also adapted to test when the subject retreats to a safe distance from the LEDs to activate the LEDs again. In this regard, when the LEDs have been turned off for a predetermined delay period, the device controller 120 is configured to reactivate the LEDs during a test period during which one or more test images are captured by the camera 106. The test period may be a fixed time, such as 0.2 seconds, 0.5 seconds, or 1 second, or a predetermined number of image frames, such as 10 frames, 20 frames, 25 frames, or 30 frames. The controller 112 is configured to execute the method 700 during the test period to determine whether the brightness of the image captured by the previously deactivated LEDs has subsequently increased (indicating that the subject is no longer blocking the LEDs). The device controller 120 is further configured to keep the LEDs activated if the brightness measurement of the test image is equal to or greater than a predetermined brightness threshold, and to deactivate the infrared light source otherwise.

[0076] Although labelled as "test images", images obtained during the test period may be used in the driver monitoring procedure if the relevant characteristics of the driver can be suitably distinguished.

[0077] The above-described system allows for monitoring of the proximity between the subject and the LEDs (by a proxy measurement of image brightness), and feedback control is provided to device controller 120 to limit or disable the output power of the LEDs based on detecting that the LEDs are an unsafe distance from the driver. The present invention may be used in combination with other LED power control systems / techniques, such as those described in WO 2015 / 031942 to Edwards, entitled "System and Method for Low Power Eye Tracking," and WO 2016 / 191827 to Edwards, entitled "Protection System for Infrared Light Sources," both of which are assigned to Seeing Machines LLC.

[0078] In operation, the method 700 is performed continuously or at predetermined intervals such that the driver is continually monitored while remaining safe from exposure to excessive infrared radiation.

[0079] In addition to providing safety benefits, the present invention also has application in creating lighting diagnostic functions for driver monitoring systems. In these applications, the present invention allows the system to detect if infrared light is being "blocked" by an obstacle at close range, which can (1) improving tracking performance in such conditions by assisting driver monitoring systems in modifying strobe lighting patterns to use only unlocked lights for tracking purposes; and (2) If the blockage persists for a significant period of time (e.g., 10 minutes), then assume that the IR light itself has some damage or defect that prevents it from outputting light. It is useful for

[0080] <<Further embodiments>> Many other methods can be used for occlusion detection, including methods that extend to single infrared source occlusion detection.

[0081] A simple example of a desired setup is shown diagrammatically at 90 in Figure 9, where a user 91 is imaged by an imaging camera 92 relying on a single infrared imaging emitter 93. A second person 95 is dangerously close to the emitter 93 such that their eyes 94 are too close to the light source. A second embodiment is designed to detect such partial occlusion cases, where the emitter is partially occluded.

[0082] In further alternative embodiments, one or more cameras and one or more IR emitting sources may be provided.

[0083] Further embodiments use an "IR occlusion detection algorithm" and a high level state machine that implements eye safety mechanisms. The system utilizes the concept of a "low energy" IR light source mode, where the IR light source is controlled such that the human eye can be exposed indefinitely without biological damage, in compliance with IEC-62471. Many techniques can be used. For example, time domain modulation of the transmit intensity of an IR light source is used to distinguish the IR light source component from the sun (or any other source) via matched filtering of the modulation pattern at the receiver (image sensor).

[0084] In a further refinement, the IR blockage detection algorithm can operate across a series of image regions, applying matched filtering of intensity over time to each image region. The algorithm then assumes that the IR light source is blocked by the eye if any of the image regions show no detection of the modulation pattern. In one arrangement, the IR light source may be placed into a low power mode when deemed blocked.

[0085] This arrangement provides the ability to protect eye safety when using only a single IR light source and not a second proximity sensor.

[0086] Generally, the intensity and activation of each IR light can be controlled, therefore in the most common designs only a single IR light is used (single dark or light pupil).

[0087] Typically, a captured image contains light from an IR emitting source in combination with other uncontrolled light sources such as the sun, ambient light, etc. To determine if there is an obstacle in close proximity to the IR light, an occlusion detection algorithm must detect if an IR light component is present in the image. To deal with partial occlusions, the image can be divided into regions and the IR light component is detected in each region.

[0088] If no IR light component is detected in a predetermined number of regions, a possible occlusion can be flagged.

[0089] To detect occlusion, one can rely on the effectiveness of the specular reflection versus time. If the effectiveness is higher than an experimentally determined threshold, it is determined that the IR light is not blocked. If the effectiveness is below the threshold, it is determined that the IR light is blocked (thus entering safety mode). The determination can be made over a period that may allow for no reflection being detected due to occlusion, posture, etc., but that is also within the hazardous exposure period of, for example, 8 seconds.

[0090] In one embodiment, a coded intensity pattern can be provided. For example, a 10% intensity variation pseudo-random illumination variation pattern is applied every frame. The known IR light output intensity is passed to a recurrent neural network (RNN) alongside intensity values ​​derived from a grid of image regions. The RNN effectively correlates changes in image intensity to changes in IR light intensity. The RNN is trained on the truth of pod occlusion, It is easily obtained.

[0091] In a further alternative, other coded activation patterns may be used, similar to coded intensity, but switching the IR light off only occasionally, as determined by intensity versus time.

[0092] In a further alternative, a structured light pattern can be used: the light source can be controlled to emit a detectable low-contrast pattern in the resulting image: if the pattern is not visible, it can be determined that the IR light is occluded, regardless of whether there is enough ambient light to produce a well-exposed image.

[0093] To separate the IR light component (signal) from other light components (noise), the IR light intensity can be intentionally modulated with a known pattern. This modulation can be in the time domain, the spatial domain, or both. The known pattern can be detected in each image region using a matched filter on the pixel intensity information.

[0094] Different modulation embodiments can be used, for example turning off the light source for one frame per second, applying a sinusoidal or square wave variation to the light intensity, applying a known pseudo-random modulation to the light intensity, etc.

[0095] While detection can rely on matched filters, other techniques such as recurrent neural networks can be used.

[0096] If an occlusion is detected, the system can transition to a lower power state to reduce the risk of damage to nearby observers. <<Interpretation>> The term "infrared" is used throughout this description and specification. Within the scope of this specification, infrared refers to the general infrared region of the electromagnetic spectrum, including near infrared, infrared and far infrared frequencies or light waves.

[0097] Unless specifically stated otherwise, and as will be apparent from the discussion that follows, discussions throughout this specification utilizing terms such as "processing," "computing," "calculating," "determining," "analyzing," and the like will be understood to refer to the operation and / or processing of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical quantities, such as electronic quantities, into other data similarly represented as physical quantities.

[0098] Similarly, the terms "controller" or "processor" may refer to any device or part of a device that processes electronic data, e.g., from registers and / or memory, and converts the electronic data into other electronic data, which may be stored, e.g., in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0099] References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of the phrases "in one embodiment," "some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, as would be apparent to one of ordinary skill in the art from this disclosure.

[0100] As used herein, unless otherwise specified, the use of ordinal adjectives such as "first," "second," "third," etc. to describe a common object is intended merely to indicate reference to different instances of a similar object, and is not intended to imply that the described objects must be in a particular order, temporally, spatially, sequentially, or in any other way.

[0101] In the following claims and in the description of this specification, any of the terms comprising, consisting of, or comprising are open terms meaning to include at least the element / feature that follows it and not to exclude others. Thus, when used in the claims, the term comprising should not be interpreted as being limited to the means or elements or steps that follow it. For example, the scope of an expression of a device comprising A and B should not be limited to a device consisting of only elements A and B. Any of the terms comprising or comprising as used in this specification are also open terms meaning to include at least the element / feature that follows it and not to exclude others. Thus, comprising is synonymous with comprising and means comprising.

[0102] In the above description of exemplary embodiments of the present disclosure, it should be understood that various features of the present disclosure may be combined in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and facilitating understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present disclosure.

[0103] Furthermore, some embodiments described herein include some features included in other embodiments but not other features. However, combinations of features of different embodiments are intended to form different embodiments within the scope of the disclosure, as would be understood by one of ordinary skill in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0104] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this specification.

[0105] Similarly, it is noted that the term coupled, when used in the claims, should not be interpreted as being limited to only direct connections. The terms "coupled" and "connected" may be used along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression that device A is coupled to device B should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. It means that there is a path between the output of device A and the input of device B, which may be a path involving other devices or means. "Coupled" may mean that two or more elements are in direct physical, electrical, or optical contact, or that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0106] The embodiments described herein are intended to include any adaptations or variations of the present invention. Although the invention has been depicted and described with reference to specific exemplary embodiments, it will be understood by those skilled in the art that additional embodiments can be readily envisioned that are within the scope of the invention.

Claims

1. A surveillance system for protecting a subject from exposure to harmful infrared light, the system comprising: one or more infrared light sources for continuously illuminating a subject; a camera that captures an image of the object while the object is illuminated by one of the one or more infrared light sources; a processor for processing the captured image to determine a luminance measurement of the captured image; a control device for controlling an output power of the one or more infrared light sources in response to the luminance measurements; In response to the processor detecting a luminance measurement of the captured image below a predetermined luminance threshold, the controller is configured to turn off or reduce an output illumination intensity of an infrared light source of the one or more infrared light sources determined to cause a reduction in the luminance measurement; In response to the processor detecting a luminance measurement of the image above the predetermined luminance threshold, the controller is configured to maintain an output illumination intensity of the one or more infrared light sources; A surveillance system wherein a brightness measurement of the captured image below the predetermined brightness threshold indicates that the object is too close to the one or more infrared light sources or that the object is occluded by the object in the captured image.

2. The surveillance system of claim 1 , wherein the brightness measure is an average pixel intensity of the image.

3. The surveillance system of claim 1 , wherein the brightness measure is an average pixel intensity of a subset of pixels of the image.

4. The surveillance system of claim 1, wherein the brightness measurements are determined for a pixel region of the image, the pixel region of the image corresponding to a face, an eye, or both eyes of the subject.

5. The surveillance system of claim 1 , wherein the brightness measurement is a comparison of brightness between two or more images.

6. The surveillance system of claim 1 , wherein the brightness measurement is a comparison of brightness between successive images.

7. The surveillance system of claim 1 , wherein the brightness measurement is a comparison of brightness between two images captured during illumination by a common infrared light source.

8. The surveillance system of claim 7 , wherein the comparison comprises a pixel-by-pixel comparison of pixel intensities of two or more images.

9. A surveillance system according to claim 5 , wherein the comparison comprises comparing average pixel intensities of two or more images.

10. 10. The surveillance system of claim 1, wherein when the infrared light source is turned off for a predetermined delay period, the controller is configured to restart the infrared light source during a test period in which one or more test images are captured, and the controller is further configured to keep the infrared light source activated if a brightness measurement of the test image is greater than or equal to a predetermined brightness threshold, and to not activate the infrared light source otherwise.

11. 11. A surveillance system according to claim 1, wherein at least one of the light sources is positioned at an angle of greater than 3.2 degrees from the camera when viewed along an optical axis from the subject.

12. 12. The monitoring system of claim 1, wherein in response to detecting a luminance measurement below a predetermined luminance threshold, the controller reduces the output illumination intensity and increases the illumination period of the infrared light source based on a determination of radiation safety to personnel.

13. 13. The monitoring system of claim 1, wherein the processor is configured to detect a current level of the brightness measurement from a plurality of predetermined levels, and in response, the controller is configured to set an output illumination intensity of one of the infrared light sources to a value corresponding to the current level.

14. A surveillance system according to any one of claims 1 to 13, mounted within a cabin of a vehicle, the subject being a driver of the vehicle.

15. A method for controlling a system of two or more infrared light sources to protect a subject from exposure to harmful infrared radiation, the method comprising: i. sequentially illuminating a subject from two or more spaced infrared light sources; ii. capturing an image of the object while the object is illuminated by one of the infrared light sources; iii. processing the captured image to determine a luminance measurement of the image; iv. comparing said luminance measurement to a predetermined luminance threshold; and controlling an output power of the infrared light sources in response to the comparison of step iv, wherein in response to detecting a luminance measurement below the predetermined luminance threshold, an output illumination intensity of one of the infrared light sources is reduced or set to zero; controlling the output power of the infrared light source includes maintaining the output illumination intensity of the infrared light source when the brightness measurement is determined to be above the predetermined brightness threshold; A method in which brightness measurements of the image below the predetermined brightness threshold indicate that the object is too close to at least one of the infrared light sources or that the object is occluded by an object in the image.

16. The method of claim 15, wherein the brightness measurements include measurements of average pixel intensity for each image.

17. The method of claim 15, wherein the brightness measurements are determined for pixel regions of the image, the pixel regions of the image corresponding to a face, an eye, or both eyes of the subject.

18. The method of claim 15, wherein the brightness measurement is a comparison of brightness between two or more images.

19. The method of claim 15, wherein the brightness measurement is a comparison of brightness between successive images.

20. The method of claim 15, wherein the brightness measurement is a comparison of brightness between two images captured while illuminated by a common infrared light source.

Images