Method for adapting the scanning rate of a vehicle passenger observation camera

Abstract

Title: Method for Adapting the Scan Rate of a Vehicle Passenger Observation Camera. A method (30) for adapting the scan rate of a vehicle occupant observation camera (24) consists of: - providing observation data from the observation camera (28), - recognizing a blink event of the vehicle occupant (24) by processing the observation data, * recognizing the beginning and end of the blink event, - determining an adaptation of the observation camera's (28) scan rate based on the recognized blink event, * increasing the scan rate if the beginning of a blink event of the vehicle occupant (24) is recognized, and * reducing the scan rate when the end of the blink event of the vehicle occupant (24) is recognized. Figure 1

Description

Title of the invention: Method for adapting the scanning rate of a vehicle passenger observation camera FIELD OF THE INVENTION

[0001] The present invention relates to a method for adapting the scanning rate of a vehicle passenger observation camera. STATE OF THE ART

[0002] According to document DE 10 2014 214 352 A1, a method and a device for managing a vehicle occupant observation system are known.

[0003] DISCLOSURE AND ADVANTAGES OF THE INVENTION

[0004] The present invention relates to a method for adapting a scanning rate of a vehicle occupant according to the following steps consisting of: providing observation data from the observation camera, recognizing a blinking event of the vehicle occupant by using the observation data, recognizing the start and end of the blinking event, determining an adaptation of the scanning rate of the observation camera depending on the recognized blinking event, increasing the scanning rate if a start of a blinking event of the vehicle occupant is recognized and reducing the scanning rate when the end of the blinking event of the vehicle occupant is recognized.

[0005] The method has the advantage that adapting the scanning rate, in particular by reducing the scanning rate, reduces the energy input for the camera. The scanning rate can advantageously be reduced if a high scanning rate is not required. Conversely, the scanning rate can be increased if this is desirable and, for example, necessary for observing the occupant. Reducing the energy input of the camera and other components, for example, the control device, improves the thermal characteristics of the components. In other words, the heating of the camera and other components can be reduced, i.e., overheating and temperature problems are advantageously avoided. This advantageously increases the service life of the components. This is advantageous, in particular in the case of compact designs or high demands on the environment, i.e., the surrounding components.This ensures and improves the functionality of the observation device, i.e. the camera.

[0006] By increasing the scanning rate by recognizing a blinking event, optimal observation of the occupant is enabled, which maintains observation performance. The advantage is to reduce the scanning rate so as not to develop temperature issues and to maintain performance by increasing the slew rate at certain times so that there is no loss of performance.

[0007] According to a development of the method, a step of observing the occupant with the observation camera is provided to determine observation data. This makes it possible to generate observation data in a particularly safe and reliable manner. In addition, reliable blinking events of the occupant are also determined.

[0008] According to another feature, the method comprises a step of adapting the scanning rate of the observation camera as a function of the determined adaptation of the scanning rate of the camera. This makes it possible to adapt the scanning rate of the observation camera in particular in a safe and / or reliable manner. It is thus possible in particular to reduce the power supply, for example, to the camera or to other components.

[0009] According to an advantageous development, the start of the occupant blinking event is recognized if the difference between the opening of an occupant eyelid between two times exceeds a threshold. Furthermore, a first opening of an occupant eyelid at a first time and a second opening of the occupant eyelid at a second time can be determined and the start of an occupant blinking event can be ascertained or recognized if the difference between the first opening and the second opening is greater than a threshold. This makes it possible to determine the occupant blinking event in a particularly safe and reliable manner and to adapt the scanning rate safely.

[0010] According to a variant, a first sum of the surface area of ​​the visible sclera, the surface area of ​​the visible iris and the surface area of ​​the visible pupil of the occupant's eye is made at a first instant and the sum of the surface area of ​​the visible sclera, the surface area of ​​the visible iris and that of the visible pupil of the occupant's eye at a second instant, and the start of an occupant blinking event is recognized if the difference between the first sum and the second sum is greater than a threshold. This thus makes it possible to determine the start of an occupant blinking event in particular in a safe and reliable manner and thus to adapt the scanning rate in a safe and reliable manner.

[0011] According to a development, the vehicle occupant flashing process is learned by a machine learning method to predict the start or end of the occupant flashing event. Alternatively or additionally, the probability of the start of an occupant flashing event can be predicted with a learned algorithm and the detection rate is then increased if the probability of a flashing event is greater than a threshold. Advantageously, as a variant, the probability of the start of the occupant flashing event is captured. the vehicle occupant using a statistical method that takes into account the time relative to the previous flashing event to increase the scanning rate if the probability of a flashing event is greater than a threshold. Predicting a probability of the start of the flashing event makes it possible to increase the scanning rate if the probability of the flashing event is relatively high. This ensures coverage of the start of the flashing event in particular also by the high scanning rate. This improves the functionality of the observation device in particular for the observation of the occupant. This allows particularly safe and reliable observation of the occupant.

[0012] According to one embodiment, the scanning rate is set at a value between approximately 40 Hz and approximately 80 Hz, in particular between approximately 50 Hz and approximately 70 Hz, in particular between 55 Hz and approximately 65 Hz, in particular equal to 60 Hz for the increase if the start of an occupant flashing event is recognized. This makes it possible to have a reliable scanning rate and to observe the occupant, in particular in a safe and reliable manner, thus considerably increasing road traffic safety.

[0013] According to an advantageous development, if the end of the occupant blinking event has been recognized, the detection rate is reduced to a value between about 10 Hz and about 50 Hz, in particular between about 50 Hz and about 40 Hz, in particular between about 25 Hz and about 35 Hz and in particular of the order of 30 Hz. This ensures the reduction of the scanning rate and thus reduces the power supply to the camera and other components, thereby reducing the likelihood of overheating of the components, which ensures the functionality and reliability of the components.

[0014] The invention also relates to an observation device for applying a method for determining and adapting the scanning rate of a camera for observing a vehicle occupant.

[0015] According to one development, the observation device has an occupant observation camera. According to a variant, the observation device is an observation camera. Brief description of the drawings

[0016] The present invention will be described below in more detail with the aid of embodiments of a method for adapting the scanning rate of a camera for observing the eye of a vehicle occupant shown in the accompanying drawings in which:

[0017] [Fig-1] diagram of a vehicle equipped with a device for observing an occupant of vehicle according to an exemplary embodiment of the invention,

[0018] [Fig.2] diagram of a method according to an embodiment of the invention,

[0019] [Fig.3] diagram of a method according to an embodiment of the invention,

[0020] [Fig.4] diagram of a method according to an embodiment of the invention,

[0021] [Fig.5] diagram of eye opening as a function of time,

[0022] [Fig.6] diagram of a vehicle occupant's eye,

[0023] [Fig.7] Diagram of the eye of a vehicle occupant.

[0024] DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0025] [Fig.l] shows a vehicle 20, for example, a motor vehicle equipped with a observation device 22 for observing one or more vehicle occupants 24 or passengers 24 according to an exemplary embodiment of the invention.

[0026] The vehicle 20 has a passenger compartment 26 or interior space 26; the interior space 26 is equipped with one or more seats 27 for one or more occupants 24. The vehicle 20 has an observation device 22 for observing the passenger compartment 26 or one or more vehicle occupants 24, for example, the driver and the front passenger or one or more other passengers. The observation device 22 may also be referred to as: an observation system, a vehicle occupant observation system, or a monitoring system. The observation device 22 is designed, for example, to capture a gaze direction, a body posture, or the position of the head, face, or eyes of the vehicle occupant 24, or the state of fatigue or other vital parameters of the occupant 24. In addition, the identity of the occupant 24 may be captured.

[0027] The observation device 22 is in particular installed in the dashboard, the instrument panel, the steering wheel, the roof of the vehicle, a window, a rearview mirror or a column, for example, column A or column B of the vehicle 20 or another location of the vehicle 20.

[0028] To observe the occupant 24, the observation device 22 has a camera 28 or a shooting unit. According to one development, the observation device 22 has a lighting unit for emitting light rays, in particular infrared rays. Alternatively, the observation device 22 does not have a separate lighting unit and the passenger compartment is illuminated by ambient light.

[0029] The lighting unit is in particular oriented towards the occupant 24 and thus towards the driver's seat 27 to illuminate the occupant 24 with light rays, in particular infrared rays. In other words, the lighting unit emits light rays, in particular infrared rays, towards the occupant 24. The lighting unit is, for example, a lighting unit, a lighting element, an LED, OLED and / or a laser diode or a lighting unit, a lighting element, a light diode, an LED, OLED or a laser diode.

[0030] The image capture unit is, for example, an image capture unit such as a sensor or a camera, in particular an infrared camera module whose image capture unit is directed towards the occupant 24 and thus towards the driver's seat 27 to visually capture the occupant 24. The embodiment as an infrared camera module allows night observation without having to illuminate the occupant 24 and thus risk disturbing or dazzling him.

[0031] In other words, the observation device 22 makes it possible to observe the passenger compartment of the vehicle. The observation device 22 comprises a lighting unit for emitting light rays, for example, infrared rays towards the interior of the passenger compartment of the vehicle and a receiving unit for receiving the light rays, for example, infrared rays. Preferably, the light rays emitted by the lighting unit can be reflected onto or by the passenger compartment and the reflected rays can be directed towards the receiving unit.

[0032] Furthermore, the observation device 22 comprises a control unit or operating unit 29 or search unit 29 for controlling the lighting unit or the receiving unit or processing the data taken by the receiving unit.

[0033] The observation device 22 is notably designed to apply a method according to [Fig.2] and [Fig.3] or [Fig.4]. For example, the method according to [Fig.2], [Fig.3] or [Fig.4] can be applied by the control unit 29 or computer 29.

[0034] [Fig. 2] is a diagram of a method 30 for adjusting the scanning rate of an observation camera observing an occupant according to an exemplary embodiment of the invention. The observation camera is part of an observation device or is designed as an observation device. The observation device can be designed as the observation device of [Fig. 1] and be installed in a vehicle.

[0035] In a first step 32 of the method 30, the observation data from the observation camera are provided. The observation data are, for example, received by an observation unit of a control device.

[0036] In a first step 34 of the method 30, a blinking event of the occupant is recognized by using the observation data, by detecting a start and an end of the blinking event.

[0037] For the detection of the blinking event, the following procedure is envisaged:

[0038] The start of the blinking event of the occupant is advantageously noted if the difference in the opening of an eyelid of the occupant between two instants is greater than a threshold. In other words, the opening of an eyelid of the occupant at a first instant and a second opening of the eyelid of the occupant at a second instant are determined, the start of the first blinking event of the vehicle occupant being recognized if the difference between the first opening and the second opening is greater than a threshold. In other words, the difference between two different eye openings at different times and the start of a blinking event is considered or recognized if the difference is greater than a threshold.

[0039] In other words, the start of an occupant blinking event is advantageously recognized by determining a reduction in the eyelid opening of the eye. An algorithm determines the eye opening ôt at any time t if the difference in eye opening ôt0-ôti> GIGG R+ exceeds a threshold G between two times {t0 0 ,tj I to< tb it is necessary to increase the camera speed or advantageously increase the camera scanning rate. The eye opening can be interpreted both metrically or in the image plane. In the image plane one can compensate for the evaluated movement of the head between times t0 and tp It is assumed that the eyelid signal is monotonous.

[0040] The recognition of the blinking event by the opening of the eyelid is presented, for example, by the diagram of [Fig.5] or with the eye of [Fig.6].

[0041] Alternatively, the start of the blinking event is also determined by determining a sum. For example, a first sum of the area of ​​a visible sclera, the area of ​​the visible iris and the area of ​​the visible pupil of the occupant's eye at a first time and a second sum of the area of ​​the visible sclera, the area of ​​the visible iris and the area of ​​the visible pupil of the occupant's eye at a second time are determined, and the start of the blinking event of the occupant is recognized if the difference of the first sum and the second sum is greater than a threshold.

[0042] In other words, the start of the occupant blinking event is advantageously recognized by determining the surface area of ​​the visible sclera, pupil and iris. With an algorithm, the sum of the visible sclera, visible iris and visible pupil surfaces At at time t is determined. If the difference in the sum of the surfaces At0-Ati> GIGG R+ exceeds a threshold g between two times {t0,ti} 110 < tb il 0 The camera scan rate must be increased. The threshold is, for example, the value of a surface. Thus, the sums of the surfaces can be metric values ​​or interpreted in the image plane. In the image plane, the evaluated movement of the head between times t0 and tp is compensated. This assumes a monotonous surface signal.

[0043] The recognition of the blinking event by the surface is presented as an example, for the eye of [Fig.7].

[0044] According to one development, optical flow compensated for head movement in the eyelid region is used to switch the camera scan rate. Optical flow characterizes the movement between two images. The differences between a range around the eyelids at time to and a range of the eyelids at time ti can be recorded with the optical flow in a vector field. With this vector field, we know in particular how many pixels move from one image to the next. This movement makes it possible to advantageously find out how the eyelids moved and whether there was eyelid closure.

[0045] According to an advantageous development, the blinking behavior, in particular of a certain occupant of the vehicle, is learned by machine learning to predict the start or end of the occupant blinking event. For example, an occupant can be identified and for this identification of the occupant the blinking behavior is learned and then the method is called. For example, the probability of the start of an occupant blinking event can be predicted by a learning algorithm, and the scanning rate is increased if the probability of a blinking event is greater than a threshold. Alternatively or additionally, the probability of the start of an occupant blinking event is predicted by a statistical method that predicts a time relative to the previous blinking event and the detection rate is increased if the probability of the event is greater than a threshold.

[0046] In other words, using a predictor, a blinking event of a vehicle occupant is predicted. In other words, a predictor makes it possible to predict blinking events. If the probability of a blinking event is greater than a threshold, the detection rate of the camera is increased. For this, predictors based on the K1 methods and / or statistical methods are applied and which are taken into account at the time of the blinking event.

[0047] In particular, it is possible to consider recording the start of the blinking event only with a low scanning rate. In this case, the start of the blinking event can be evaluated, for example, with a parametric evaluation model. For the calibration and determination of the parametric model, past events, the subsequent course of the blinking events with a modified scanning rate and also assumptions are used, for example.

[0048] False positives may notably increase the energy input into the system components but have no other effect on the overall system.

[0049] In a third step 36 of the method 30, the scanning rate of the observation camera is adapted as a function of the received, determined blinking event and the detection rate is increased if the start of a blinking event of the occupant is recognized and the detection rate is reduced if the end of the blinking event of the occupant is recognized.

[0050] Advantageously, if a start of an occupant flashing event is recognized, the scanning rate has a value between approximately 40 Hz and approximately 80 Hz, in particular between approximately 45 Hz and approximately 80 Hz, in particular between approximately 45 Hz and 75 Hz, in particular between approximately 50 Hz and 70 Hz and in particular between approximately 55 Hz and approximately 65 Hz and in particular it is equal to approximately 60 Hz.

[0051] Preferably, if the end of the occupant flashing event has been recognized, the scanning rate is reduced to a value between approximately 10 Hz and 50 Hz, between approximately 10 Hz and approximately 45 Hz, between approximately 15 Hz and approximately 45 Hz, in particular between approximately 20 Hz and approximately 40 Hz, in particular between approximately 25 Hz and approximately 35 Hz, in particular of the order of 30 Hz.

[0052] In other words, the method 30 is a method for adapting the scanning rate depending on the behavior of the eyelid. The target is the reduction of the average scanning rate to reduce the energy input to the system components. To observe the driver, one or more observation cameras can be used with a variable scanning rate, for example, depending on a parameter such as, for example, the duration, the start or the end of the blinking event. Advantageously, with the method, the camera can operate with a lower scanning rate and increase it only during a determined or recognized blinking event. In other words, the blinking event is classified.

[0053] [Fig. 3] is a diagram of a method 30 according to one embodiment of the invention for adapting the scanning rate of an observation camera that observes a vehicle occupant. The observation camera may be part of an observation device or be the observation device. The observation device may be implemented like the observation device of [Fig. 1] and / or be installed in a vehicle.

[0054] The method 30 comprises in particular the steps of the method 30 of [Fig. 2], namely the first step 32, the second step 34 and the third step 36. The method 30 according to [Fig. 3] is supplemented by a further step 38. In this fourth step 38, the scanning rate of the observation camera is adapted as a function of the determined adaptation of the scanning rate of the observation camera. Advantageously, the scanning rate of the observation camera is increased when the start of an occupant blinking event is detected and the scanning rate is reduced when the end of the occupant blinking event is recognized. The fourth step 38 applies in particular after the third step 36 consisting of determining the adaptation of the scanning rate.

[0055] [Fig. 4] is a diagram of a method 30 for adapting the scanning rate of a camera to the observation of an occupant according to an exemplary embodiment of the present invention. The observation camera may be a part of the observation device or be the observation device. The observation device may be embodied as the observation device of [Fig. 1] or be installed in a vehicle.

[0056] The method 30 comprises in particular the steps of the method 30 of [Fig. 2], namely a first step 32, a second step 34 and a third step 36. According to a development, the variant 30 further comprises the fourth step 38 of the method 30 of [Fig. 3]. The method 30 of [Fig. 4] has a fifth step 40. In this fifth step 40, the occupant is observed with the observation camera to determine observation data. The fifth step 40 can be done before the first step 32.

[0057] [Fig. 5] is the diagram of a diagram 42 representing the opening of an eyelid as a function of time according to an exemplary embodiment of the present invention. The x-axis 44 of the diagram 42 which represents time. The y-axis 46 of the diagram 42 which represents the opening of the eyelid in millimeters. The diagram 42 represents two functions 48, 50 of the eyelid openings as a function of time. The two functions 48, 50 are substantially parallel to the x-axis 44. For this, the scanning rate according to the brace 52 is, for example, maintained at 30 Hz. Preferably, the scanning rate is maintained at a value between approximately 10 Hz and approximately 45 Hz, in particular between approximately 20 Hz and approximately 40 Hz and in particular between approximately 25 Hz and 35 Hz and in particular it is of the order of 30 Hz.

[0058] In the area of ​​the circle 54, the start of a blinking event is recognized. This recognition is done according to the method of FIGS. 2, 3 or 4. The opening of the eye and thus the functions 48, 50 arrive at a lower value. This results in a lowering of the opening of the eyelid. In the area of ​​the descent, the detection rate is raised according to the brace 56, for example, to 60 Hz. Advantageously, the scanning rate can be set to a value between approximately 45 Hz and approximately 80 Hz, in particular between approximately 50 Hz and approximately 70 Hz, in particular between approximately 55 Hz and 65 Hz, and in particular it is of the order of 60 Hz if the start of a blinking event of the occupant is recognized.

[0059] In other words, diagram 42 represents the eyelid opening as a function of time, in mm, during a blinking event. The diagram shows two examples of tracing the eyelid opening during a blinking event from the development of an eyelid opening reference system. The method makes it possible to recognize the start of a characteristic blink from which the scanning rate of the camera is increased.

[0060] [Fig. 6] shows the diagram of an eye 58 of a vehicle occupant. The method or algorithm recognizes the lower eyelid 60 and the upper eyelid 62. The maximum distance between the lower eyelid 60 and the upper eyelid 62 is measured, which corresponds to the eyelid opening 64, in particular at the first instant. The eyelid opening 64 can be expressed in millimeters. This determination is made at several instants. The start of a blinking event is recognized by determining the opening of eyelids, for example, by the method according to figures 2, 3 or 4.

[0061] [Fig. 7] is a diagram of an occupant's eye 58. The eye 58 has a visible sclera surface 66, a visible iris surface 68, and a visible pupil surface 70. These surfaces are added together to obtain the total surface area or sum of the surfaces A. The onset of a blinking event is recognized by determining the sum of the surfaces, for example, using the method of FIGS. 2, 3, or 4.

[0062] NOMENCLATURE OF MAIN ELEMENTS

[0063] 20 Vehicle

[0064] 22 Observation device

[0065] 24 Vehicle Passenger / Occupant

[0066] 26 Passenger compartment

[0067] 27 Seat

[0068] 29 Control unit, operating unit, computing unit

[0069] 30 Method

[0070] 32, 34, 36, 38, 40 Process steps [0071 ] 42 Diagram

[0072] 44 X Axis

[0073] 46 Y Axis

[0074] 48, 50 Functions

[0075] 52 Accolade

[0076] 54 Circle

[0077] 56 Accolade

[0078] 58 Eye

[0079] 60 Lower eyelid

[0080] 62 Upper eyelid

[0081] 64 Opening between the eyelids

[0082] 66 Sclera

[0083] 68 Iris

[0084] 70 Pupil

Claims

Claims

1. Method (30) for adapting a scanning rate of an observation camera (28) of a vehicle occupant (24) according to the following steps: - providing observation data of the observation camera (28), - recognizing a blinking event of the vehicle occupant (24) by using the observation data, according to which * recognizing the start and the end of the blinking event, - determining an adaptation of the scanning rate of the observation camera (28) depending on the recognized blinking event, * increasing the scanning rate if a start of a blinking event of the vehicle occupant (24) is recognized, and * reducing the scanning rate when the end of the blinking event of the vehicle occupant (24) is recognized.

2. A method (30) according to claim 1, comprising a step of: - observing the vehicle occupant (24) with the observation camera (28) to determine observation data.

3. Method (30) according to one of the preceding claims, consisting of: - adapting the scanning rate of the observation camera (28) as a function of the determined adaptation of the scanning rate of the observation camera (28).

4. Method (30) according to one of the preceding claims, according to which the start of the blinking event of the vehicle occupant (24) is recognized if the difference between the opening of an eyelid of the occupant (24) between two instants is greater than a threshold.

5. Method (30) according to one of the preceding claims, according to which a first opening of an eyelid of the occupant (24) is determined at a first instant and a second opening of the eyelid of the occupant (24) is determined at a second instant, - the start of the occupant flashing event (24) being recognized when the difference between the first opening and the second opening exceeds a threshold.

6. Method (30) according to one of the preceding claims, according to which a first sum of a surface of a visible sclera of a surface of the visible iris and a surface of the visible pupil of the eye of the occupant (24) is determined at a first instant and a second sum of the surface of the visible sclera, the surface of the visible iris and the surface of the visible pupil of the eye of the occupant (24) at a second instant, * the start of a blinking event of the occupant (24) is recognized if the difference of the first sum and the second sum is greater than a threshold.

7. A method (30) according to one of the preceding claims, wherein the blinking behavior of the occupant (24) is learned by a machine learning method to predict the start or end of the blinking event of the occupant (24).

8. Method (30) according to one of the preceding claims, according to which the probability of the start of a blinking event of the occupant (24) is predicted with a learning algorithm, * the scanning rate is increased if the probability of a blinking event is greater than a threshold.

9. Method (30) according to one of the preceding claims, according to which the probability of the start of a blinking event of the occupant (24) is predicted with a statistical method which takes into account the previous blinking event, * the scanning rate is increased if the probability of a blinking event is greater than a threshold.

10. Method (30) according to one of the preceding claims, according to which - if a start of a blinking event of the occupant (24) is recognized, the scanning rate is increased to a value between approximately 40 Hz and approximately 80 Hz, in particular included between approximately 50 Hz and 70 Hz, in particular between approximately 55 Hz and 65 Hz, in particular approximately 60 Hz.

11. Method (30) according to one of the preceding claims, according to which if the end of the occupant flashing event (24) is recognized, the scanning rate is reduced to a value between approximately 10 Hz and approximately 50 Hz, in particular between approximately 20 Hz and approximately 40 Hz, in particular between approximately 25 Hz and approximately 35 Hz, in particular approximately equal to 30 Hz.

12. Observation device (22) for applying a method (30) according to one of claims 1 to 11.

13. An observation device (22) according to claim 12, comprising an observation camera (28) for observing a vehicle occupant (24).

Images