Method for operating a driver assistance system for traffic light visualisation, and motor vehicle comprising such a driver assistance system

Abstract

The invention relates to a method (10) for operating a driver assistance system (7) for traffic light visualisation by means of a display (6) in a motor vehicle (1), the method comprising the following steps: - determining (11) a viewing direction of a driver (3) of the motor vehicle (1), - determining (12) an image of a corneal reflection at at least one eye of the driver (3), - determining (13) traffic lights and the positions thereof in the image of the corneal reflection, and calculating associated traffic-light positions in vehicle surroundings (9) located in front of the motor vehicle (1), - evaluating (14) the determined viewing direction with regard to the calculated traffic-light positions in the vehicle surroundings (9), and monitoring whether, how often, and / or for how long each traffic light is being looked at by the driver (3), - evaluating (15) the traffic lights viewed by the driver (3) and identifying the traffic light that controls the roadway on which the motor vehicle (1) is located, - tracking (16) the identified traffic light in the image of the corneal reflection, and - activating (17) the traffic-light visualisation on the display (6) as soon as the identified traffic light moves out of the image of the corneal reflection.

Description

[0001] Mercedes-Benz Group AG

[0002] Method for operating a driver assistance system for traffic light visualization and motor vehicle with such a driver assistance system

[0003] The present invention relates to a method for operating a driver assistance system for traffic light visualization by means of a display in a motor vehicle. The invention also relates to a motor vehicle equipped with a driver assistance system.

[0004] Modern vehicles can be equipped with a driver assistance system that displays traffic light information on a screen. Traffic lights are particularly useful in urban traffic, where they help control intersections. An intersection is where two or more lanes meet, crossing or intersecting each other. Traffic lights have become the standard for controlling right-of-way and ensuring orderly traffic flow. Modern vehicles are often equipped with a front-facing camera that captures an image or video stream of the area in front of the vehicle. As the vehicle approaches or stops at an intersection, the driver assistance system can display the current traffic light status on a suitable screen inside the vehicle.This is particularly advantageous when the vehicle is close to a traffic light, making it difficult for the driver (male / female / diverse) to see the light. However, it has been shown that such driver assistance systems also display traffic light information even when the driver has a clear view of the light and is actively observing it. This can be perceived as distracting or confusing, and therefore disadvantageous or even disruptive, especially when combined with a head-up display that can be used for traffic light visualization.

[0005] There is therefore a need to modify a driver assistance system for traffic light visualization in such a way that the traffic light visualization is not always performed or displayed on the respective screen, but only when this results in increased vehicle safety. The present invention therefore addresses the problem of providing a solution for a driver assistance system for traffic light visualization via a display in a motor vehicle, or for a motor vehicle equipped with such a driver assistance system, that is characterized by increased comfort and / or reduced distraction, thereby increasing vehicle safety.

[0006] This problem is solved according to the invention by the subject matter of the independent claims. Advantageous embodiments are the subject matter of the dependent claims.

[0007] The invention is based on the general concept of activating the traffic light visualization on the display only when a traffic light controlling a lane on which the vehicle is traveling or stationary disappears from the driver's field of vision. The invention is based on the consideration that the relevant traffic light is recognizable and noticed by the driver as long as it remains within their field of vision. By activating the traffic light visualization on the display as soon as the relevant traffic light disappears from the driver's field of vision, the driver's attention can be subtly drawn to the relevant traffic light, thus mitigating a potentially dangerous situation and increasing vehicle safety.

[0008] The invention evaluates the driver's field of vision using an image of a corneal endoscopy taken from at least one of the driver's eyes. This corneal endoscopy image reflects the vehicle's surroundings currently visible to the driver, i.e., the driver's field of vision. Traffic lights located within the driver's field of vision are also included in the corneal endoscopy image and can be identified using appropriate algorithms, as well as their position within the area in front of the vehicle. Furthermore, the driver's current gaze direction can be determined relatively accurately, making it possible to identify whether the driver is looking at a traffic light, how often, and for how long.If there are multiple traffic lights in the area in front of the vehicle, it can also be checked whether the driver pays more attention to one of these lights than to others. This makes it possible to filter out the relevant traffic light from among several – that is, the one that controls the lane the vehicle is on. The identified traffic light can then be tracked in the corneal scintigraphy image. As long as this identified traffic light is in the corneal scintigraphy image, it is also in the driver's field of vision, so it can be assumed that the driver is paying attention to it. However, as soon as the identified traffic light leaves the corneal scintigraphy image, it is no longer in the driver's field of vision and therefore can no longer be noticed by the driver.In this case, the traffic light visualization on the display is deactivated, allowing the driver to immediately recognize and observe the relevant traffic light on the display.

[0009] Specifically, the invention proposes a method for operating a driver assistance system for traffic light activation via a display in a motor vehicle. This method involves determining the driver's gaze direction, acquiring a corneal endoscopy image of at least one of the driver's eyes, identifying the traffic lights and their positions within the corneal endoscopy image, and calculating the corresponding traffic light positions in the vehicle's surroundings. Furthermore, the determined gaze direction is evaluated with respect to the calculated traffic light positions in the vehicle's surroundings, observing whether, how often, and / or for how long the driver views each traffic light. Subsequently, the traffic lights viewed by the driver can be evaluated to identify the traffic light controlling the lane on which the motor vehicle is located.The identified traffic light is tracked in the corneal endoscopy image, and the traffic light visualization on the display is activated as soon as the identified traffic light leaves the corneal endoscopy image.

[0010] According to an advantageous embodiment, the traffic light that the driver looks at most frequently and / or for the longest time can be identified as the one that controls the lane on which the vehicle is located. This embodiment is based on the consideration that a reasonably acting driver intuitively pays more attention to the traffic light that is most relevant to him or his vehicle, i.e., the one that controls the lane on which the vehicle is located. The driver pays increased attention to a traffic light if he looks at it particularly often and for a comparatively long time, in any case more often and / or for a longer time than all other traffic lights that may also be located in the vicinity of the vehicle.

[0011] According to an advantageous embodiment, the evaluation of the driver's gaze direction with respect to the calculated traffic light positions in the vehicle's vicinity, the observation of whether, how often, and / or for how long the respective traffic light is viewed by the driver, the selection of the traffic lights viewed by the driver, and the identification of the traffic light controlling the lane on which the vehicle is located can be continuously repeated. This allows for situations where the driver changes lanes relatively late when approaching an intersection, meaning that a different traffic light may be responsible for controlling the changed lane in the short term.The method presented here therefore also reacts to a short-term lane change, without requiring or even needing to be present any sensors that can also detect such a lane change, or any navigation system that can also detect a lane change.

[0012] In another advantageous embodiment, the determination of traffic lights and their positions in the corneal scintigraphy image and the calculation of the corresponding traffic light positions in the vehicle's surroundings are performed using templates, i.e., predefined patterns for a multitude of different traffic light-controlled intersection situations. These templates can be stored, for example, in a suitable memory, particularly a database, making it especially easy to determine typical traffic light positions depending on the specific intersection situation. The use of such templates requires comparatively little computing power and can therefore be implemented quickly and easily. Alternatively or additionally to such templates, a data-driven artificial intelligence fed with a large amount of data is also conceivable.Furthermore, a color space can also be observed in which the typical colors of traffic lights, especially green, yellow and red, are taken into account.

[0013] According to a particularly advantageous embodiment, the traffic light visualization on the display can be deactivated as soon as the identified traffic light reappears in the corneal scintigraphy image. In other words, the traffic light visualization on the display is only active as long as the traffic light relevant to the vehicle is not in the corneal scintigraphy image, i.e., not in the driver's field of vision. This significantly improves comfort and thus vehicle safety.

[0014] Determining the driver's gaze direction can be achieved, for example, using a driver observation camera. A method for determining the driver's gaze direction using a driver observation camera is known from DE 10 2014 100 352 A1. The corneal examination image of the respective eye can also be generated using a driver observation camera. An advantageous embodiment is one in which the driver's gaze direction is determined using a driver observation camera that is also used to generate the corneal examination image of the respective eye. Accordingly, instead of using two separate cameras, one and the same driver observation camera is employed, which simplifies the implementation of the method.

[0015] In another advantageous embodiment, a front-facing camera of the vehicle can be used to generate a video stream of the vehicle's surroundings, which is then displayed on the respective screen when the traffic light visualization is activated. The front camera can have a wide-angle lens for this purpose. The respective screen is a display device that can be configured, for example, as a monitor located on the vehicle's instrument panel in the vehicle interior, such as in the area of ​​a center console. It is also conceivable to configure the display as a head-up display, so that the information to be displayed, in particular the traffic light visualization, is projected onto the windshield.

[0016] In the present context, a "configuration" is synonymous with a "design" and / or "setup" and / or "programming", so that the phrase "configured so that" is synonymous with the phrase "designed and / or set up and / or programmed so that".

[0017] According to another embodiment, the identified traffic light is only tracked as long as it is located in the vehicle's immediate vicinity. The moving vehicle passes through the intersection and thus moves past the relevant traffic light, at which point the traffic light is no longer relevant and further monitoring of it is no longer necessary.

[0018] A motor vehicle according to the invention has a passenger compartment in which a driver can sit. The vehicle is equipped with a driver observation camera for monitoring the driver seated in the passenger compartment. The vehicle is also equipped with a front camera for generating a video stream of the vehicle's surroundings in front of it. A display, which can be configured as a monitor or a head-up display, is located in the passenger compartment. Furthermore, the vehicle is equipped with a driver assistance system that is coupled to the front camera and the display and is configured to visualize traffic lights on the display. The vehicle is also equipped with a control unit that is coupled to the driver observation camera and the driver assistance system and is configured to carry out the method as described above.The control unit can be part of the driver assistance system, meaning it can be at least partially integrated into the hardware and / or at least partially implemented in the software.

[0019] Further important features and advantages of the invention will become apparent from the dependent claims, the drawings and the associated description of the figures based on the drawings.

[0020] It is understood that the features mentioned above and those to be explained below can be used not only in the combinations specified, but also in other combinations or individually, without departing from the scope of the invention as defined by the claims. Components of a higher-level unit, such as a device, apparatus, or arrangement, mentioned above and those to be mentioned below, which are designated separately, can form separate parts or components of this unit or be integral areas or sections of this unit, even if this is depicted differently in the drawings.

[0021] Preferred embodiments of the invention are shown in the drawings and are explained in more detail in the following description, wherein identical reference numerals refer to identical or similar or functionally identical components.

[0022] They show, schematically,

[0023] Figure 1 shows a highly simplified, circuit diagram-like schematic representation of a motor vehicle with a driver assistance system.

[0024] Figure 2 is a flowchart illustrating a procedure for operating the driver assistance system.

[0025] As shown in Figure 1, a motor vehicle 1, which may be a passenger car, a truck, or another commercial vehicle, comprises a vehicle interior 2 in which a driver 3 can sit. The motor vehicle 1 is equipped with a driver observation camera 4, a front camera 5, at least one display 6, a driver assistance system 7, and a control unit 8. The driver observation camera 4 is configured to monitor the driver 3 sitting in the vehicle interior 2. The front camera 5 is configured to generate a video stream of the vehicle environment 9 in front of the motor vehicle 1. The respective display 6 is configured to visually display information to the driver 3. In the example shown in Figure 1, two different displays 6 are indicated purely by way of example: a display 6.1 designed as a monitor and a display 6.2 designed as a head-up display.The driver assistance system 7 is coupled to the front camera 5 and the respective display 6. Corresponding lines for transmitting signals and / or data are shown, but not specified in detail. The driver assistance system 7 is configured to visualize traffic lights on the respective display 6. For this purpose, the video stream generated by the front camera 5 is displayed on the respective display 6, thus showing the traffic lights present in the vehicle's surroundings 9 in front of the vehicle 1 on the display 6 and making them recognizable to the driver 3. The control unit 8 is coupled to the driver monitoring camera 4 and to the driver assistance system 7. Corresponding lines for signal and / or data transmission are shown, but not specified in detail. The control unit 8 is configured to carry out a procedure 10 symbolized by a flowchart in Figure 2.

[0026] Method 10, as shown in Figure 2, serves to operate the driver assistance system 7 for traffic light visualization via the respective display 6 in the motor vehicle 1. Method 10 comprises steps 11 to 18, which are explained in more detail below. In step 11, the driver's 3 gaze direction is determined. In step 12, which can be performed simultaneously, before, or after step 11, an image of a corneal smear is obtained from at least one eye of the driver 3. In step 13, traffic lights and their positions in the corneal smear image are determined. Furthermore, in step 13, the traffic light positions in the vehicle environment 9 in front of the motor vehicle 1, corresponding to the traffic lights determined in the corneal smear image, are calculated.For a simplified calculation of the traffic light positions in the vehicle's surroundings 9, it can be useful to rectify the corneal smear image in step 12 or step 13. Subsequently, the traffic lights and their positions can be determined in the rectified corneal smear image. Furthermore, the corresponding traffic light positions in the vehicle's surroundings 9 can be calculated based on the positions of the traffic lights in the rectified corneal smear image. The cornea of ​​the human eye is a spherically curved body, i.e., a three-dimensional structure. The corneal smear image, on the other hand, is two-dimensional and therefore inevitably distorted, which makes converting a spatial position in the distorted image to an actual spatial position in the vehicle's surroundings extremely complex. Rectifying the captured image significantly simplifies the assignment of a position in the rectified image to a position in the vehicle's surroundings.

[0027] In step 14, the gaze direction determined in step 11 is evaluated with regard to the traffic light positions calculated in step 13 in the vehicle's surroundings 9. It is observed whether, how often, and / or for how long the respective traffic light is viewed by driver 3. In the subsequent step 15, the traffic lights viewed by driver 3 are evaluated. Furthermore, the traffic light controlling a lane on which vehicle 1 is located is identified—that is, the lane on which vehicle 1 is driving or stationary. In step 16, the identified traffic light is tracked within the corneal scintigraphy image. In step 17, the traffic light visualization is activated on the respective display 6 as soon as the identified traffic light leaves the corneal scintigraphy image. A boundary can be assigned to the cornea in the image for this purpose. As soon as the tracked traffic light crosses this boundary, it is considered to be outside the corneal scintigraphy image.In the following step 18, the traffic light visualization is deactivated again in the respective display 6. The conditions for deactivating the traffic light visualization in step 18 are explained further below.

[0028] The identification of the relevant traffic light in step 15, i.e., the traffic light that controls the lane on which vehicle 1 is located, is based on the consideration that the relevant traffic light is viewed most frequently and / or for the longest time by driver 3 when vehicle 1 approaches a traffic light-controlled intersection. Accordingly, the traffic light that driver 3 views most frequently and / or for the longest time is identified as the traffic light that controls the lane on which vehicle 1 is located.The evaluation of the direction of gaze with regard to the calculated traffic light positions in the vehicle environment 9, carried out in step 14, and the observation of whether, how often and / or how long the respective traffic light is viewed by the driver 3, as well as the evaluation of the traffic lights viewed by the driver and the identification of the relevant traffic light carried out in step 15, is constantly repeated, which in particular allows a change of the relevant traffic light to be detected quickly and reliably.

[0029] The determination of the traffic lights and their positions in the corneal scintigraphy image, as performed in step 13, and the calculation of the corresponding traffic light positions in the vehicle's surroundings 9, can be conveniently carried out using templates that are provided for a large number of different traffic light-controlled intersection situations. The templates can be stored in a corresponding memory in the control unit 8. It is also conceivable that the control unit 8 has access to a data storage device (not shown here) in which the templates are stored.

[0030] The deactivation of the traffic light visualization in the respective display 6, as performed in step 18, can be carried out, for example, as soon as the identified traffic light reappears in the corneal scintigraphy image or crosses the aforementioned boundary inwards again. If the relevant traffic light is located at the edge of the corneal scintigraphy image and therefore repeatedly disappears from and reappears in the corneal scintigraphy image, the traffic light visualization will be alternately activated and deactivated. This ultimately focuses the driver's attention strongly on this traffic light, allowing the driver to intuitively adjust their field of vision accordingly, so that the identified traffic light then remains permanently in the corneal scintigraphy image and the traffic light visualization can be deactivated.

[0031] According to an advantageous embodiment, the driver's 3 gaze direction is determined by means of the driver observation camera 4. For this purpose, corresponding algorithms for evaluating the driver's 3 face and / or eyes can be stored in the control unit 8, enabling the determination of the gaze direction. Advantageously, the same driver observation camera 4 is also used to capture the image of the corneal reflection in the respective eye of the driver 3. The driver observation camera 4 has a correspondingly high resolution. As the vehicle 1 travels on the respective lane, it approaches the intersection and can then proceed through it. Accordingly, the vehicle 1 then also passes the traffic light relevant for lane control, meaning that the traffic light is no longer located in the vehicle's immediate vicinity 9 in front of the vehicle 1.Therefore, the identified traffic light in the corneal scintigraphy image is only tracked as long as it remains within the vehicle environment 9 in front of vehicle 1. Once vehicle 1 has passed the traffic light, it becomes irrelevant for its further journey. In step 18, the traffic light visualization is also deactivated if the identified traffic light leaves the vehicle environment 9 in front of vehicle 1. In other words, the traffic light visualization on display 6 is also deactivated when the tracking of the relevant traffic light in the corneal scintigraphy image is terminated.

[0032] Procedure 10 is repeated continuously, preferably at a predetermined cycle time. It is clear that the detection of traffic lights in the corneal endoscopy image, as performed in step 13, is only possible if there are actually traffic lights in the vehicle environment 9 in front of the vehicle 1.

Claims

1. Mercedes-Benz Group AG Patent claims 1. Method (10) for operating a driver assistance system (7) for traffic light visualization by means of a display (6) in a motor vehicle (1), comprising the following steps: - Determining (11) the direction of view of a driver (3) of the motor vehicle (1), - Obtaining (12) an image of a corneal endoscopy of at least one eye of the driver (3), - Determining (13) traffic lights and their positions in the corneal endoscopy image and calculating associated traffic light positions in a vehicle environment (9) in front of the motor vehicle (1), - Evaluating (14) the determined direction of gaze with regard to the calculated traffic light positions in the vehicle environment (9) and observing whether, how often and / or how long the respective traffic light is viewed by the driver (3), - Evaluating (15) the traffic lights viewed by the driver (3) and identifying the traffic light that controls a lane on which the motor vehicle (1) is located, - Tracking (16) the identified traffic light in the corneal endoscopy image, and - Activate (17) the traffic light visualization in the display (6) as soon as the identified traffic light leaves the corneal examination image.

2. Method (10) according to claim 1, characterized in that, - that the traffic light which is viewed most frequently and / or for the longest time by the driver (3) is identified as the traffic light which controls the roadway on which the motor vehicle (1) is located.

3. Method (10) according to claim 1 or 2, characterized in that, - that the evaluation (15) of the direction of view with regard to the calculated traffic light positions in the vehicle's surroundings (9), the observation of whether, how often and / or how long the respective traffic light is viewed by the driver (3), the evaluation of the traffic lights viewed by the driver (3) and the identification of the traffic light which controls the roadway on which the motor vehicle (1) is located are continuously repeated.

4. Method (10) according to one of the preceding claims, characterized in that - that the determination (13) of the traffic lights and their positions in the corneal endoscopy image and the calculation of the associated traffic light positions in the vehicle environment (9) in front of the motor vehicle (1) is carried out using templates that are provided for a large number of different traffic light controlled intersection situations.

5. Method (10) according to any one of the preceding claims, characterized in that - that the traffic light visualization in the display (6) is deactivated (18) as soon as the identified traffic light reappears in the corneal smear image.

6. Method (10) according to any one of the preceding claims, characterized in that - that the determination of the driver's gaze direction (3) is carried out using a driver observation camera (4), which is also used to determine the corneal reflection image of the respective eye of the driver (3).

7. Method (10) according to one of the preceding claims, characterized in that - that a video stream of the vehicle environment (9) in front of the vehicle (1) is generated by means of a front camera (5), which is displayed in the respective display (6) when traffic light visualization is activated.

8. Method (10) according to any one of the preceding claims, characterized in that - that the identified traffic light is only tracked as long as this traffic light is located in the vehicle environment (9) in front of the vehicle (1).

9. Method (10) according to claim 8, characterized in that - that the traffic light visualization in the display (6) is deactivated as soon as the identified traffic light leaves the vehicle environment (9) in front of the vehicle (1).

10. Motor vehicle (1), - with a vehicle interior (2) in which a driver (3) can sit, - with a driver observation camera (4) for observing the driver (3) sitting inside the vehicle (2), - with a front camera (5) for generating a video stream of a vehicle environment (9) in front of the motor vehicle (1), - with a display (6) arranged in the vehicle interior (2), - with a driver assistance system (7) that is coupled with the front camera (5) and with the display (6) and that is configured to visualize traffic lights on the display (6), - with a control unit (8) that is coupled to the driver observation camera (4) and to the driver assistance system (7) and that is configured to perform the method (10) according to one of the preceding claims.

Images