Method for adjusting sampling rate of monitoring camera for monitoring vehicle occupant

Abstract

To propose a method (30) for adjusting a sampling rate of a monitoring camera (28) for monitoring a vehicle occupant (24).SOLUTION: A method (30) includes the steps of: providing monitoring data of a monitoring camera (28); identifying a blink event of a vehicle occupant (24) by evaluation of the monitoring data, and identifying the start and end of the blink event; determining adjustment of a sampling rate of the monitoring camera (28) depending on the identified blink event, while increasing the sampling rate when the start of the blink event of the vehicle occupant (24) is identified, and reducing the sampling rate when the end of the blink event of the vehicle occupant (24) is identified.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Prior Art From DE 10 2014 214 352 A1 a method and a device for operating an occupant monitoring system are known. [Prior art documents] [Patent documents]

[0002] [Patent Document 1] DE 102014214352 Summary of the Invention [Means for solving the problem]

[0003] Disclosure of the Invention 1. A method for adjusting a sampling rate of a surveillance camera for monitoring vehicle occupants, comprising: Providing surveillance data from a surveillance camera; identifying a blink event of the vehicle occupant by evaluating the monitored data, wherein a start and an end of the blink event are identified; determining an adjustment of a sampling rate of the surveillance camera in dependence on the identified blink event, where if a start of a vehicle occupant blink event is identified, the sampling rate is increased, and if an end of a vehicle occupant blink event is identified, the sampling rate is decreased; A method is proposed which includes:

[0004] The advantage of the method is that the energy input to the camera can be reduced by adjusting the sampling rate, in particular by reducing the sampling rate. Advantageously, the sampling rate can be reduced if an increase in the sampling rate is not necessary. Advantageously, the sampling rate can be increased if this is desired, for example necessary for monitoring vehicle occupants. Due to the reduction in the energy input to the camera and further components, for example the control device, the thermal properties of the components can be significantly improved. In other words, overheating of the camera and further components can be reduced, so that they are not particularly overheated and no temperature problems arise. Advantageously, this can increase or extend the life of the components. This is an advantage in particular in the case of compact construction types and / or when the requirements of the environment or surrounding components are high. This can ensure or guarantee the function of the monitoring device or camera.

[0005] Increasing the sampling rate upon identifying a blink event can allow optimal monitoring of the vehicle occupant, thereby maintaining monitoring performance. The advantage is therefore that the sampling rate can be reduced, thereby not causing temperature issues, and at a given point in time, increasing the sampling rate can continue to maintain performance, thereby not causing performance degradation.

[0006] In an exemplary embodiment, the method may further include monitoring the vehicle occupant with a surveillance camera to determine the surveillance data, which may be generated particularly reliably and / or reliably, and which may further enable a reliable and / or reliable determination of eye blink events of the vehicle occupant.

[0007] Furthermore, the method may in particular comprise a step of adjusting the sampling rate of the surveillance camera depending on the determined adjustment of the sampling rate of the surveillance camera, whereby the sampling rate of the surveillance camera can be adjusted in a particularly reliable and / or reliable manner, whereby in particular, for example, the energy input to the camera and / or further components can be reduced.

[0008] In a preferred embodiment, the start of a blink event of the vehicle occupant can be identified if a difference in the opening degree of the eyelids of the vehicle occupant between two time points is greater than a threshold value. Furthermore, a first opening degree of the eyelids of the vehicle occupant at a first time point and a second opening degree of the eyelids of the vehicle occupant at a second time point can be determined, whereby the start of a blink event of the vehicle occupant can be identified if a difference between the first opening degree and the second opening degree is greater than a threshold value. This allows a particularly reliable and / or highly reliable determination of the start of a blink event of the vehicle occupant, whereby the sampling rate can be adjusted particularly reliably and / or highly reliable.

[0009] In an alternative or additional embodiment, a first sum of the visible sclera area, the visible iris area and the visible pupil area of ​​the vehicle occupant's eye at a first time point and a second sum of the visible sclera area, the visible iris area and the visible pupil area of ​​the vehicle occupant's eye at a second time point may be determined, where the start of a blink event of the vehicle occupant is identified if a difference between the first sum and the second sum is greater than a threshold value. This allows a particularly reliable and / or reliable determination of the start of a blink event of the vehicle occupant, whereby the sampling rate can be adjusted particularly reliably and / or reliably.

[0010] In a further development, the blinking behavior of the vehicle occupant can be learned using machine learning methods to predict the start and / or end of a blinking event of the vehicle occupant. Alternatively or additionally, in particular the probability of the start of a blinking event of the vehicle occupant can be predicted using a learned algorithm, where the sampling rate is increased if the probability for a blinking event is greater than a threshold. Advantageously, alternatively or additionally, the probability of the start of a blinking event of the vehicle occupant can be predicted using a statistical method that takes into account the time from a preceding blinking event, where the sampling rate is increased if the probability for a blinking event is greater than a threshold. Due to the prediction of the probability of the start of a blinking event, the sampling rate can be increased, in particular if the probability for a blinking event is relatively high. This makes it possible to ensure that the start of a blinking event can be captured even with a particularly high sampling rate. This can improve the functionality of the monitoring device, in particular the monitoring of the vehicle occupant. This allows vehicle occupants to be monitored in a particularly reliable and / or reliable manner.

[0011] In an exemplary embodiment, the sampling rate may be increased to a value between about 40 and about 80 Hz, in particular between about 50 and about 70 Hz, in particular between about 55 and about 65 Hz, in particular to about 60 Hz, when the onset of a blinking event of the vehicle occupant is identified. This may allow a particularly reliable sampling rate, whereby the vehicle occupant may be monitored particularly reliably and / or reliably, which may increase safety in road traffic, in particular.

[0012] In a preferred embodiment, the sampling rate can be reduced to a value between about 10 Hz and about 50 Hz, in particular between about 20 Hz and about 40 Hz, in particular between about 25 Hz and about 35 Hz, in particular about 30 Hz, when the end of the blinking event of the vehicle occupant is identified. This ensures that the sampling rate is reduced, in particular so that the energy input to the camera or further components is reduced. This ensures that the probability of overheating of the components can be reduced, thereby ensuring their functionality and / or longevity.

[0013] Furthermore, a surveillance device is proposed, which is configured to execute the method for determining an adjustment of the sampling rate of a surveillance camera for monitoring a vehicle occupant.

[0014] In a further development, the monitoring device can comprise a surveillance camera for monitoring the vehicle occupants.In an alternative embodiment, the monitoring device can be configured as a surveillance camera.

[0015] Embodiments of the invention are illustrated in the drawings and are explained in more detail in the following specification, in which the same reference numerals are used for elements which act similarly and which are illustrated in different drawings, and where a repeated description of these elements is omitted. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic diagram illustrating a vehicle equipped with a monitoring device for monitoring a vehicle occupant according to an embodiment of the present invention; [Diagram 2] 1 is a schematic diagram illustrating a method according to an embodiment of the present invention; [Diagram 3] 1 is a schematic diagram illustrating a method according to an embodiment of the present invention; [Figure 4] 1 is a schematic diagram illustrating a method according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a schematic diagram illustrating a graph of eyelid opening over time according to an embodiment of the present invention. [Figure 6] 1 is a schematic diagram illustrating an eye of a vehicle occupant according to one embodiment of the present invention. [Figure 7] 1 is a schematic diagram illustrating an eye of a vehicle occupant according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] EMBODIMENTS OF THE PRESENTINVENTION Fig. 1 shows a schematic diagram of a vehicle 20, for example a motor vehicle, e.g. a private car, equipped with a monitoring device 22 for monitoring one or more vehicle occupants 24 or passengers 24 according to an embodiment of the invention. The vehicle 20 has, among other things, an interior space 26 or a cabin space 26, in which, among other things, one or more seats 27 for one or more vehicle occupants 24 are arranged. Furthermore, the vehicle 20 comprises a monitoring device 22 for monitoring the cabin space 26 and / or the one or more vehicle occupants 24, for example a driver and / or a front passenger seat occupant and / or one or more further passengers. The monitoring device 22 may also be referred to as a monitoring system, a driver monitoring system, an occupant monitoring system and / or an observation system. The monitoring device 22 may, for example, be configured to capture the gaze direction, the body posture and / or the head or face or eye position of the vehicle occupant 24, or the fatigue state and / or other vital values ​​of the vehicle occupant 24. Additionally, for example, the identity of the vehicle occupant 24 may be captured.

[0018] The monitoring device 22 may be located, among other places, in the dashboard, in the instrument panel, in the steering wheel, on the vehicle roof, on the windshield, in the rearview mirror, or on a pillar, such as the A-pillar and / or B-pillar of the vehicle 20, or elsewhere in the vehicle 20.

[0019] To monitor the vehicle occupants 24, the monitoring device 22 may have a camera 28 or a recording unit. In one development, the monitoring device 22 may have a lighting unit for emitting a light beam, in particular an infrared beam. Alternatively, the monitoring device 22 does not have a separate lighting unit and the interior space or the vehicle occupants can also be illuminated using ambient light.

[0020] The lighting unit may in particular be oriented in the direction of the vehicle occupant 24, and thus in the direction of the vehicle seat 27, in order to illuminate the vehicle occupant 24 with a light beam, in particular an infrared beam. In other words, the lighting unit may be used to illuminate a light beam, in particular an infrared beam, in the direction of the vehicle occupant 24. The lighting unit may for example be configured as and / or comprise a light emitting unit, a light emitting element, a light emitting diode, an LED, an OLED and / or a laser diode.

[0021] The recording unit may be configured, for example, as an imaging unit, for example as a sensor or as a camera, in particular as an infrared camera module, where the recording unit is oriented in the direction of the vehicle occupant 24 and thus in the direction of the vehicle seat 27 in order to visually capture the vehicle occupant 24. The configuration as an infrared camera module makes it possible to carry out monitoring also at night, in particular without being hindered or dazzled by bright illumination of the vehicle occupant 24.

[0022] In other words, the monitoring device 22 is configured to monitor the interior space of the vehicle. The monitoring device 22 may include an illumination unit for emitting a light beam, e.g. an infrared beam, in the direction of the interior space and a recording unit for recording the light beam, e.g. the infrared beam. Preferably, the light beam emitted by the illumination unit may be reflectable in or within the interior space, where the reflected light beam may be deflected in the direction of the recording unit.

[0023] Furthermore, the monitoring device 22 may comprise a control unit 29 or an evaluation unit 29 or a calculation unit 29 for driving and controlling the lighting unit and / or the recording unit and / or for processing the data recorded by means of the recording unit.

[0024] The monitoring device 22 is in particular configured to execute the method according to Figure 2, Figure 3 and / or Figure 4. For example, the method according to Figure 2, Figure 3 and / or Figure 4 can be executed on or by means of the control unit 29 or the computing unit 29.

[0025] Fig. 2 shows a schematic diagram of a method 30 for adjusting the sampling rate of a surveillance camera for monitoring vehicle occupants according to an embodiment of the invention. This surveillance camera may be part of or configured as a surveillance device. The surveillance device may be configured according to the surveillance device according to Fig. 1 and / or may be located in a vehicle.

[0026] In a first step 32 of the method 30, surveillance data of a surveillance camera are provided. These surveillance data can be received, for example, by means of a receiving unit of the control device.

[0027] In a second step 34 of the method 30, vehicle occupant blink events are identified by evaluation of the monitored data, where the start and end of the blink event are identified.

[0028] For blink event detection, the following procedure is considered here.

[0029] The start of a vehicle occupant's blink event can preferably be identified when the difference in the opening degree of the vehicle occupant's eyelids between two time points is greater than a threshold value. In other words, the first opening degree of the vehicle occupant's eyelids at the first time point and the second opening degree of the vehicle occupant's eyelids at the second time point can be obtained, where the start of the vehicle occupant's blink event is identified when the difference between the first opening degree and the second opening degree is greater than the threshold value. In other words, the difference between the opening degrees of two different eyelids at different time points is obtained, where the start of the blink event is identified when the difference is greater than the threshold value.

[0030] In other words, the start of a vehicle occupant's blink event can preferably be identified by obtaining a decrease in the opening degree of the eyelid. The algorithm can specify the opening degree δ of the eyelid at each time point t. t When the difference in the opening degree of the eyelid between two time points {t0, t1}|t0 < t1 exceeds the threshold value ε,

Number

[0031] The identification of the blink event using the opening degree of the eyelid can be represented, for example, using the graph according to FIG. 5 and / or using the eye according to FIG. 6.

[0032] Alternatively or additionally, the start of a blink event can also be determined using a total calculation. For example, a first total of the visible sclera area, visible iris area, and visible pupil area of the vehicle occupant's eye at a first point in time and a second total of the visible sclera area, visible iris area, and visible pupil area of the vehicle occupant's eye at a second point in time can be determined, where the start of the blink event of the vehicle occupant is identified when the difference between the first total and the second total is greater than a threshold value.

[0033] In other words, the start of the blink event of the vehicle occupant can preferably be identified by determining the areas of the visible sclera, visible pupil, and visible iris. By using an algorithm, the total A of the visible sclera area, visible iris area, and visible pupil area at each time point t t can be specified. If the difference in the total area between two time points {t0, t1}|t0 < t1 exceeds a threshold value ε, [Number] the sampling rate of the camera should be increased or can be increased. The threshold value can be, for example, formed as a value of the area. In that case, the total area can be interpreted both metrically and in the image plane. In the image plane, compensation for the head movement estimated between time points t0 and t1 can be carried out. In that case, conditions regarding the monotonicity of the area signal are considered.

[0034] The identification of the blink event using the area can be represented, for example, by using the eye according to FIG. 7.

[0035] In one development, the optical flow compensated for the head movement in the eyelid area can be used to switch the sampling rate of the camera. The optical flow describes the movement between two images. The difference between the area around the eyelid at time t0 and the area of ​​the eyelid at time t1 can be stored in a vector field by the optical flow. This vector field shows, in particular, how the pixels move from one image to the next. Based on this movement, it is preferably possible to verify how the eyelids move and whether they are closed.

[0036] In a preferred embodiment, the blink behavior of a vehicle occupant, particularly a specific vehicle occupant, can be learned using machine learning methods to predict the start and / or end of a blink event of the vehicle occupant. For example, a vehicle occupant can be identified, in which case the blink behavior of the identified vehicle occupant can be learned accordingly and can be called up later during the method. For example, the probability of the start of a blink event of the vehicle occupant can be predicted using a learned algorithm, in which the sampling rate is increased if the probability for a blink event is greater than a threshold. Alternatively or additionally, the probability of the start of a blink event of the vehicle occupant can be predicted using a statistical method that takes into account the time since the previous blink event, in which the sampling rate is increased if the probability for a blink event is greater than a threshold.

[0037] In other words, the predictor can be used to predict a blink event of a vehicle occupant. In other words, the blink event is predictable by the predictor. If the probability for a blink event exceeds a threshold, the sampling rate of the camera can be increased. The predictor can then be based on AI and / or statistical methods that take into account the time since the previous blink event.

[0038] In particular, it is conceivable that the beginning of blinking event can only be recorded at a lower sampling rate.In such a case, for example, can use parameterizable estimation model to estimate the beginning of the blinking event sequence.For example, can also use past blinking events, the subsequent sequence of blinking events at the switched sampling frequency and assumptions, etc., for calibration and determination of model parameters.

[0039] False positives increase the energy input to a particular system component but have no further effect on the system as a whole.

[0040] In a third step 36 of the method 30, an adjustment of the sampling rate of the surveillance camera is determined depending on the identified blink event, where if the start of a vehicle occupant blink event is identified, the sampling rate is increased, and where if the end of the vehicle occupant blink event is identified, the sampling rate is decreased.

[0041] Preferably, the sampling rate can be increased to a value between about 40 Hz and about 80 Hz, particularly between about 45 Hz and about 80 Hz, particularly between about 45 Hz and about 75 Hz, particularly between about 50 Hz and about 70 Hz, particularly between about 55 Hz and about 65 Hz, particularly about 60 Hz, when the start of a vehicle occupant blink event is identified. Preferably, the sampling rate can be reduced to a value between about 10 Hz and about 50 Hz, between about 10 Hz and about 45 Hz, between about 15 Hz and about 45 Hz, particularly between about 20 Hz and about 40 Hz, particularly between about 25 Hz and about 35 Hz, particularly about 30 Hz, when the end of a vehicle occupant blink event is detected.

[0042] In other words, the method 30 is configured as a method for adjusting the image sampling rate depending on the eyelid behavior. The purpose is in particular to reduce the average image sampling rate in order to reduce the energy input to the system components. In order to monitor the driver, one or more driver monitoring cameras can be operated at a variable sampling rate, which allows for determining parameters such as the duration, start or end of a blink event. Advantageously, the method 30 allows the camera to be operated at a lower frame rate, where the frame rate is increased only during a desired or identified blink event. In other words, a classification of the blink event is possible.

[0043] Fig. 3 shows a schematic diagram of a method 30 for adjusting the sampling rate of a surveillance camera for monitoring vehicle occupants according to an embodiment of the invention. The surveillance camera may be part of or configured as a surveillance device. The surveillance device may be configured according to the surveillance device according to Fig. 1 and / or may be located in a vehicle.

[0044] The method 30 includes in particular the steps of the method 30 according to Fig. 2, thus the first step 32, the second step 34 and the third step 36. The method 30 according to Fig. 3 includes a further fourth step 38. In the fourth step 38 of the method 30, the sampling rate of the surveillance camera is adjusted depending on the determined adjustment of the sampling rate of the surveillance camera. Preferably, the sampling rate of the surveillance camera is increased when the start of a blink event of the vehicle occupant is identified and is reduced when the end of the blink event of the vehicle occupant is identified. The fourth step 38 can in particular be performed after the third step 36 of determining the adjustment of the sampling rate.

[0045] Fig. 4 shows a schematic diagram of a method 30 for adjusting the sampling rate of a surveillance camera for monitoring vehicle occupants according to an embodiment of the invention. This surveillance camera may be part of or configured as a surveillance device. The surveillance device may be configured according to the surveillance device according to Fig. 1 and / or may be located in a vehicle.

[0046] The method 30 comprises in particular the steps of the method 30 according to Fig. 2, thus a first step 32, a second step 34 and a third step 36. In a development, the method 30 may additionally comprise a fourth step 38 of the method 30 according to Fig. 3. The method 30 according to Fig. 4 comprises a further fifth step 40. In the fifth step 40 of the method 30, the vehicle occupants are monitored by means of a surveillance camera in order to determine surveillance data. This fifth step 40 can in particular be carried out before the first step 32.

[0047] Fig. 5 shows a schematic diagram of a graph 42 of eyelid opening over time according to an embodiment of the invention. The graph 42 has an x-axis 44 representing time. Furthermore, the graph 42 has a y-axis 46 representing eyelid opening in millimeters. In the graph 42, two functions 48, 50 of eyelid opening over time are represented. Both functions 48, 50 run at least initially approximately parallel to the x-axis 44. Here, the sampling rate is maintained, for example, at about 30 Hz according to the curly bracket 52. Advantageously, the sampling rate can be maintained at a value between about 10 Hz and about 45 Hz, in particular between about 20 Hz and about 40 Hz, in particular between about 25 Hz and about 35 Hz, in particular about 30 Hz.

[0048] In the region of the circle 54, the beginning of a blink event is identified. This identification can be performed, for example, according to the method according to Fig. 2, Fig. 3 and / or Fig. 4. The eyelid opening, and thus the functions 48, 50, drop to a low value. This results in a reduction in the eyelid opening. In the region of this reduction, the sampling rate is raised or increased, for example to about 60 Hz, according to the curly bracket 56. Advantageously, the sampling rate can be increased to a value between about 45 Hz and about 80 Hz, in particular between about 50 Hz and about 70 Hz, in particular between about 55 Hz and about 65 Hz, in particular to about 60 Hz, when the beginning of a blink event of the vehicle occupant is identified.

[0049] In other words, graph 42 shows eyelid opening in mm with respect to time during a blink event. In other words, the graph shows two examples of the progression of eyelid opening during a blink event from the development of a reference system for eyelid opening. The method can identify the start of a blink event, in which case the frame rate of the camera is increased.

[0050] Fig. 6 shows a schematic diagram of an eye 58 of a vehicle occupant according to an embodiment of the invention. The method or algorithm identifies a lower eyelid 60 and an upper eyelid 62. The maximum distance between the lower eyelid 60 and the upper eyelid 62 is measured. This results in an eyelid opening 64, particularly at a first time point. This eyelid opening 64 can be given, for example, in millimeters. This calculation can be performed at several time points. The start of a blink event using the calculation of the eyelid opening can be identified, for example, using the method according to Fig. 2, Fig. 3 and / or Fig. 4.

[0051] 7 shows a schematic diagram of a vehicle occupant's eye 58 according to one embodiment of the present invention. The eye 58 has an area of ​​a visible sclera 66, an area of ​​a visible iris 68 and an area of ​​a visible pupil 70. These areas may be summed, for example, into a total area or sum of areas A. The start of a blink event may be identified using the calculation of the sum of areas, for example, using the methods according to FIG. 2, FIG. 3 and / or FIG. 4.

Claims

1. A method (30) for adjusting a sampling rate of a surveillance camera (28) for monitoring a vehicle occupant (24), comprising the steps of: providing surveillance data from said surveillance camera (28); identifying blink events of the vehicle occupant (24) by evaluating the monitored data, wherein a start and an end of the blink event are identified; determining an adjustment of the sampling rate of the surveillance camera (28) depending on the identified blink event, where if a start of the blink event of the vehicle occupant (24) is identified, the sampling rate is increased, and if an end of the blink event of the vehicle occupant (24) is identified, the sampling rate is decreased; A method (30) comprising:

2. 2. The method of claim 1, comprising monitoring the vehicle occupant (24) with the surveillance camera (28) to determine the surveillance data.

3. 3. The method of claim 1, further comprising adjusting the sampling rate of the surveillance camera (28) depending on the determined adjustment of the sampling rate of the surveillance camera (28).

4. 4. The method of claim 1, wherein the start of a blink event of the vehicle occupant is identified when a difference in eyelid opening of the vehicle occupant between two time points is greater than a threshold value.

5. 5. The method of claim 1, wherein a first degree of eyelid opening of the vehicle occupant at a first time point and a second degree of eyelid opening of the vehicle occupant at a second time point are determined, and the start of the blink event of the vehicle occupant is identified when a difference between the first degree of opening and the second degree of opening is greater than a threshold value.

6. 6. The method of claim 1, further comprising determining a first sum of a visible sclera area, a visible iris area, and a visible pupil area of ​​the eye of the vehicle occupant at a first time point and a second sum of the visible sclera area, the visible iris area, and the visible pupil area of ​​the eye of the vehicle occupant at a second time point, and identifying the start of a blink event of the vehicle occupant when a difference between the first sum and the second sum is greater than a threshold value.

7. 7. The method (30) of claim 1, wherein the blink behavior of the vehicle occupant (24) is learned using machine learning methods to predict the onset and / or end of a blink event of the vehicle occupant (24).

8. 8. The method (30) of claim 1, wherein the probability of the onset of a blink event of the vehicle occupant (24) is predicted using a learned algorithm, and the sampling rate is increased if the probability for a blink event is greater than a threshold.

9. 9. The method (30) of claim 1, wherein the probability of the onset of a blink event of the vehicle occupant (24) is predicted using a statistical method taking into account the time since a previous blink event, and the sampling rate is increased if the probability for a blink event is greater than a threshold.

10. 10. The method (30) according to any one of claims 1 to 9, wherein the sampling rate is increased to a value between about 40 Hz and about 80 Hz, in particular between about 50 Hz and about 70 Hz, in particular between about 55 Hz and about 65 Hz, in particular to about 60 Hz, when the start of a blink event of the vehicle occupant (24) is identified.

11. 11. The method (30) according to any one of claims 1 to 10, wherein the sampling rate is reduced to a value between about 10 Hz and about 50 Hz, in particular between about 20 Hz and about 40 Hz, in particular between about 25 Hz and about 35 Hz, in particular to about 30 Hz, when the end of a blink event of the vehicle occupant (24) is identified.

12. A monitoring device (22), configured to perform a method (30) according to any one of claims 1 to 11.

13. 13. The surveillance device (22) of claim 12, comprising a surveillance camera (28) for monitoring a vehicle occupant (24).

Images