Method for operating a driver assistance system for traffic light visualization and motor vehicle with such a driver assistance system

Abstract

The invention relates to a method (10) for operating a driver assistance system (7) for traffic light visualization by means of a display (6) in a motor vehicle (1), comprising the following steps: - determining (11) a viewing direction of a driver (3) of the motor vehicle (1), - determining (12) an image of a corneal reflection on at least one eye of the driver (3), - Determining (13) traffic lights and their positions in the corneal endoscopy image and calculating associated traffic light positions in a vehicle environment (9) in front of the motor vehicle (1), - Evaluating (14) the determined viewing direction with regard to the calculated traffic light positions in the vehicle environment (9) and observing whether, how often and / or how long the respective traffic light is viewed by the driver (3), - evaluating (15) the traffic lights viewed by the driver (3) and identifying the traffic light that controls a lane on which the motor vehicle (1) is located, - Tracking (16) the identified traffic light in the corneal endoscopy image, - Activation (17) of the traffic light visualization in the display (6) as soon as the identified traffic light leaves the corneal endoscopy image.

Description

[0001] The present invention relates to a method for operating a driver assistance system for traffic light visualization using a display in a motor vehicle. The invention also relates to a motor vehicle equipped with a driver assistance system.

[0002] Modern motor vehicles can be equipped with a driver assistance system that enables traffic light visualization via a display. Intersection situations, particularly in urban traffic, can be controlled using traffic lights. At an intersection, two or more lanes meet and cross each other. Traffic lights have been established to control the lanes or the respective intersection in order to ensure orderly right-of-way. Modern motor vehicles are often equipped with a front camera that generates an image or video stream of the vehicle's surroundings in front of the vehicle. As the vehicle approaches an intersection or stops at an intersection, the driver assistance system can ensure that the current traffic light situation is visualized on a suitable display in the vehicle interior.This is particularly advantageous when the vehicle is parked close to a traffic light, making it difficult for the driver (male / female / diverse) to see it. However, it has been shown that such driver assistance systems still visualize the traffic light even when the respective traffic light is clearly visible to the driver and is also being observed by the driver themselves. This can be perceived by the driver as distracting or confusing, and thus disadvantageous or even annoying, particularly in conjunction with a head-up display that can be used for traffic light visualization. In this context, document DE 10 2008 038 831 A1 shows a method and a device for automatically guiding the gaze of a driver of a vehicle, and document DE 10 2015 005 222 A1 describes a method for determining at least one traffic light that is relevant for a vehicle.

[0003] There is therefore a need to modify a driver assistance system for traffic light visualization in such a way that the traffic light visualization is not carried out or shown on the respective display in every case, but only when this can increase vehicle safety.

[0004] The present invention therefore deals with the problem of showing a way for a driver assistance system for traffic light visualization by means of a display in a motor vehicle or for a motor vehicle equipped with such a driver assistance system, which is characterized by increased comfort and / or reduced distraction in order to thus increase vehicle safety.

[0005] This problem is solved according to the invention by the subject matter of the independent claims. Advantageous embodiments are the subject matter of the dependent claims.

[0006] The invention is based on the general idea of activating the traffic light visualization on the display only when a traffic light controlling a lane on which the vehicle is traveling or stationary disappears from the driver's field of vision. The invention is based on the idea that the relevant traffic light is recognizable and observed by the driver as long as it is in the driver's field of vision. By activating the traffic light visualization on the display as soon as the relevant traffic light disappears from the driver's field of vision, the driver's attention can be unobtrusively directed to the relevant traffic light, which defuses a potentially dangerous situation and thus increases vehicle safety.

[0007] The invention evaluates the field of vision based on a corneal endoscopy image of at least one of the driver's eyes. This corneal endoscopy image reflects the vehicle surroundings currently visible to the driver in front of the vehicle, i.e., the driver's field of vision. Traffic lights located in the driver's field of vision are also included in the corneal endoscopy image and can be identified using appropriate algorithms and assigned to their position in the environment in front of the vehicle. Furthermore, the driver's current line of sight can be determined relatively accurately, making it possible to determine whether the driver is looking at a traffic light, how often they do so, and for how long.If there are several traffic lights in the area ahead of the vehicle, it can also be checked whether the driver is paying more attention to one of these traffic lights than to others. This makes it possible to filter out the relevant traffic light from several traffic lights, i.e. the traffic light that controls the lane on which the vehicle is located. The traffic light identified in this way can now be tracked in the corneal endoscopy image. As long as this identified traffic light is in the corneal endoscopy image, it is also in the driver's field of vision, so it is assumed that the driver is paying attention to the traffic light. However, as soon as the identified traffic light leaves the corneal endoscopy image, the traffic light is no longer in the driver's field of vision and can therefore no longer be noticed by the driver.In this case, the traffic light visualization on the display is deactivated, allowing the driver to immediately recognize and observe the relevant traffic light on the display.

[0008] In detail, the invention proposes a method for operating a driver assistance system for traffic light activation by means of a display in a motor vehicle, in which method a viewing direction of a driver of the motor vehicle is determined, in which an image of a corneal endoscopy of at least one eye of the driver is determined, and at the traffic lights and their positions in the corneal endoscopy image are determined and associated traffic light positions in a vehicle environment in front of the motor vehicle are calculated. Furthermore, the determined viewing direction is evaluated with regard to the calculated traffic light positions in the vehicle environment, wherein it is observed whether, how often and / or how long the respective traffic light is viewed by the driver. Subsequently, the traffic lights viewed by the driver can be evaluated in order to identify the traffic light that controls a roadway on which the motor vehicle is located.The traffic light identified in this way is tracked in the corneal endoscopy image and the traffic light visualization in the display is activated as soon as the identified traffic light leaves the corneal endoscopy image.

[0009] According to an advantageous embodiment, the traffic light that the driver observes most frequently and / or for the longest time can be identified as the traffic light that controls the lane on which the vehicle is located. This embodiment is based on the idea that a prudent driver intuitively pays more attention to the traffic light that has increased relevance for him or her or his vehicle, i.e., that controls the lane on which the vehicle is located. The driver pays increased attention to a traffic light if he observes it particularly frequently and for a comparatively long time, in any case more often and / or longer than all other traffic lights that may also be located in the vehicle's vicinity in front of the vehicle.

[0010] According to an advantageous embodiment, it can be provided that the evaluation of the viewing direction with regard to the calculated traffic light positions in the vehicle's surroundings, the observation of whether, how often, and / or how long the respective traffic light is viewed by the driver, the control of the traffic lights viewed by the driver, and the identification of the traffic light that controls the lane on which the motor vehicle is located are continuously repeated. This makes it possible to account for a situation in which the driver changes lanes relatively late when approaching an intersection, so that a different traffic light can temporarily be responsible for controlling the changed lane.The method presented here therefore also reacts to a short-term lane change without the need for or even the presence of sensors that can also detect such a lane change or a navigation system that can also detect a lane change.

[0011] In another advantageous embodiment, it can be provided that the determination of the traffic lights and their positions in the corneal endoscopy image and the calculation of the associated traffic light positions in the vehicle environment in front of the motor vehicle are carried out using templates that are provided for a large number of different intersection situations controlled by traffic lights. These templates can be provided, for example, in a corresponding memory, in particular in a database, which makes it particularly easy to determine typical positions of traffic lights depending on the respective intersection situation. The use of such templates requires comparatively little computing capacity and can therefore be implemented quickly and easily. Alternatively or in addition to such templates, a data-driven artificial intelligence fed with a large number of data is also conceivable.Furthermore, a color space can also be observed in which the typical colors of traffic lights, especially green, yellow and red, are taken into account.

[0012] According to a particularly advantageous embodiment, the traffic light visualization on the display can be deactivated as soon as the identified traffic light reappears in the corneal endoscopy image. In other words, the traffic light visualization on the display is only active as long as the traffic light relevant to the vehicle is not in the corneal endoscopy image, i.e., not in the driver's field of vision. This significantly improves comfort and thus vehicle safety.

[0013] The driver's line of sight can be determined, for example, using a driver observation camera. DE 10 2014 100 352 A1 discloses a method for determining the driver's line of sight using a driver observation camera. The image of the corneal reflection at the respective eye can also be generated using a driver observation camera. An embodiment is now expedient in which the driver's line of sight is determined using a driver observation camera that is also used to determine the image of the corneal reflection at the respective eye of the driver. Accordingly, one and the same driver observation camera is used here rather than two separate cameras, which simplifies the effort required to implement the method.

[0014] In another advantageous embodiment, it can be provided that a front camera of the motor vehicle is used to generate a video stream of the vehicle environment in front of the motor vehicle, which is shown on the respective display when the traffic light visualization is activated. The front camera can have a wide-angle lens for this purpose. The respective display is a display device that can be configured, for example, as a monitor located on an instrument panel of the motor vehicle in the vehicle interior, for example in the area of a center console. It is also conceivable to configure the display as a head-up display, so that the information to be displayed, in particular the traffic light visualization, is projected onto the windshield.

[0015] In the present context, a ‘configuration’ is synonymous with a ‘design’ and / or ‘arrangement’ and / or ‘programming’, so that the phrase ‘configured so that’ is synonymous with the phrase ‘designed and / or arranged and / or programmed so that’.

[0016] According to another embodiment, it can be provided that the identified traffic light is only tracked as long as this traffic light is located in the vehicle environment in front of the vehicle. The moving vehicle passes through the intersection situations and moves past the relevant traffic light accordingly, so that the traffic light in question is no longer relevant and further observation of the traffic light in question is no longer necessary.

[0017] A motor vehicle according to the invention has a vehicle interior in which a driver can sit. The motor vehicle is equipped with a driver observation camera for observing the driver sitting in the vehicle interior. The vehicle is also equipped with a front camera for generating a video stream of the vehicle environment in front of the motor vehicle. The vehicle interior contains a display that can be configured as a monitor or head-up display. Furthermore, the motor vehicle is equipped with a driver assistance system that is coupled to the front camera and the display and that is configured to visualize traffic lights on the display. Furthermore, the motor vehicle is equipped with a control device that is coupled to the driver observation camera and the driver assistance system and that is configured to carry out the method as described above.The control device can be part of the driver assistance system, i.e. at least partially integrated in hardware and / or at least partially implemented in software.

[0018] Further important features and advantages of the invention emerge from the subclaims, from the drawings and from the associated description of the figures with reference to the drawings.

[0019] It is understood that the features mentioned above and those to be explained below can be used not only in the respective combination specified, but also in other combinations or on their own, without departing from the scope of the invention as defined by the claims. Components mentioned above and to be mentioned below of a higher-level unit, such as a device, an apparatus, or an arrangement, which are designated separately, may form separate parts or components of this unit or be integral areas or sections of this unit, even if this is shown differently in the drawings.

[0020] Preferred embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description, wherein the same reference numerals refer to the same or similar or functionally identical components.

[0021] They show, schematically, Fig. 1 a highly simplified, circuit diagram-like schematic representation of a motor vehicle with a driver assistance system, Fig. 2 a flowchart illustrating a method for operating the driver assistance system.

[0022] Accordingly Fig. 1 comprises a motor vehicle 1, which may be a passenger car or a truck or other commercial vehicle, a vehicle interior 2 in which a driver 3 can sit. The motor vehicle 1 is equipped with a driver observation camera 4, a front camera 5, at least one display 6, a driver assistance system 7 and a control device 8. The driver observation camera 4 is configured to observe the driver 3 sitting in the vehicle interior 2. The front camera 5 is configured to generate a video stream of a vehicle environment 9 in front of the motor vehicle 1. The respective display 6 is configured to visually display information for the driver 3. In the example of Fig. 1, two different displays 6 are shown purely by way of example, namely a display 6 configured as a monitor 6.1 and a display 6 configured as a head-up display 6.2. The driver assistance system 7 is coupled to the front camera 5 and to the respective display 6. Corresponding lines for transmitting signals or data are shown, but not designated in more detail. The driver assistance system 7 is configured for traffic light visualization on the respective display 6. For this purpose, the video stream generated using the front camera 5 is shown on the respective display 6, whereby the traffic lights present in the vehicle environment 9 in front of the vehicle 1 are shown in the display 6 and are recognizable for the driver 3. The control device 8 is coupled to the driver observation camera 4 and to the driver assistance system 7. Corresponding lines for signal and / or data transmission are shown, but not designated in more detail. The control device 8 is configured to carry out a Fig. 2 configured by a flowchart symbolized by process 10.

[0023] Procedure 10 serves according to Fig.2 for operating the driver assistance system 7 for traffic light visualization using the respective display 6 in the motor vehicle 1. The method 10 comprises steps 11 to 18, which are explained in more detail below. In step 11, a viewing direction of the driver 3 is determined. In step 12, which can be performed simultaneously, before, or after step 11, an image of a corneal endoscopy of at least one eye of the driver 3 is determined. In step 13, traffic lights and their positions in the corneal endoscopy image are determined. In addition, in step 13, traffic light positions in the vehicle environment 9 in front of the motor vehicle 1 that correspond to the traffic lights determined in the corneal endoscopy image are calculated. For a simplified calculation of the traffic light positions in the vehicle environment 9, it may be expedient to rectify the corneal endoscopy image in step 12 or step 13.Subsequently, the traffic lights and their positions in the rectified image of the corneal endoscopy can be determined. In addition, the corresponding traffic light positions in the vehicle's surroundings 9 can be calculated based on the positions of the traffic lights in the rectified image of the corneal endoscopy. The cornea of the human eye is a spherically curved body, i.e., a three-dimensional structure. The image of the corneal endoscopy, on the other hand, is two-dimensional and therefore inevitably distorted, which makes converting a spatial position in the distorted image to an actual spatial position in the vehicle's surroundings extremely complex. Rectifying the captured image significantly simplifies the assignment of a position in the rectified image to a position in the vehicle's surroundings.

[0024] In step 14, the viewing direction determined in step 11 is evaluated with regard to the traffic light positions in the vehicle environment 9 calculated in step 13. In the process, it is observed whether, how often, and / or for how long the respective traffic light is viewed by driver 3. In the subsequent step 15, the traffic lights viewed by driver 3 are evaluated. Furthermore, the traffic light that controls a lane on which vehicle 1 is located, i.e. the lane on which vehicle 1 is traveling or stationary, is identified. In step 16, the identified traffic light is tracked in the corneal endoscopy image. In step 17, the traffic light visualization is activated in the respective display 6 as soon as the identified traffic light leaves the corneal endoscopy image. For this purpose, a boundary can be assigned to the cornea in the image. As soon as the tracked traffic light in the image exceeds this boundary, it is considered to be outside the corneal endoscopy image.In the following step 18, the traffic light visualization is deactivated again in the respective display 6. The boundary conditions for deactivating the traffic light visualization in step 18 are explained further below.

[0025] The identification of the relevant traffic light performed in step 15, i.e., the traffic light controlling the lane on which vehicle 1 is located, is based on the consideration that the relevant traffic light is observed by driver 3 for the longest time and / or most frequently when vehicle 1 approaches an intersection controlled by traffic lights. Accordingly, the traffic light observed by driver 3 for the longest time and / or most frequently is identified as the traffic light controlling the lane on which motor vehicle 1 is located.

[0026] The evaluation of the viewing direction with regard to the calculated traffic light positions in the vehicle environment 9 carried out in step 14 and the observation of whether, how often and / or how long the respective traffic light is viewed by the driver 3, as well as the evaluation of the traffic lights viewed by the driver carried out in step 15 and the identification of the relevant traffic light are continuously repeated, whereby in particular a change in the relevant traffic light can be detected quickly and reliably.

[0027] The determination of the traffic lights and their positions in the corneal endoscopy image performed in step 13, as well as the calculation of the corresponding traffic light positions in the vehicle environment 9, can expediently be carried out using templates provided for a variety of different intersection situations controlled by traffic lights. The templates can be provided in a corresponding memory in the control device 8. It is also conceivable that the control device 8 has access to a data memory (not shown in detail here) in which the templates are stored.

[0028] The deactivation of the traffic light visualization in the respective display 6 performed in step 18 can, for example, be performed as soon as the identified traffic light reappears in the corneal endoscopy image or crosses the aforementioned boundary again. If the relevant traffic light is located at the edge of the corneal endoscopy image and thus repeatedly disappears from the corneal endoscopy image and returns to the corneal endoscopy image, the traffic light visualization is alternately activated and deactivated accordingly, which ultimately allows the driver's attention to be emphatically directed to this traffic light, causing the driver to intuitively adjust their field of vision accordingly, so that the identified traffic light then remains permanently in the corneal endoscopy image and the traffic light visualization can be deactivated accordingly.

[0029] According to an advantageous embodiment, the driver's 3 line of sight is determined using the driver observation camera 4. For this purpose, appropriate algorithms for evaluating a face and / or the eyes of the driver 3 can be stored, in particular, in the control device 8, which algorithms enable the determination of the line of sight. The same driver observation camera 4 is expediently also used to capture the image of the corneal reflection at the respective eye of the driver 3. The driver observation camera 4 has a correspondingly high resolution.

[0030] If vehicle 1 is traveling on the respective lane, vehicle 1 approaches the respective intersection situation and can then finally drive through it. Accordingly, vehicle 1 then moves past the traffic light relevant for steering the lane, as a result of which the relevant traffic light is then no longer located in the vehicle environment 9 in front of vehicle 1. It is therefore expedient to track the identified traffic light in the corneal endoscopy image only as long as this traffic light is located in the vehicle environment 9 in front of vehicle 1. As soon as vehicle 1 has passed the traffic light in question, this traffic light is irrelevant for the further travel of vehicle 1. In step 18, the traffic light visualization is also deactivated when the identified traffic light leaves the vehicle environment 9 in front of vehicle 1.In other words, the traffic light visualization in display 6 is also deactivated when tracking of the relevant traffic light in the corneal endoscopy image is stopped.

[0031] Method 10 is repeated continuously, preferably at a predetermined cycle time. It is clear that the detection of traffic lights in the corneal endoscopy image performed in step 13 is only possible if there are actually traffic lights in the vehicle environment 9 in front of vehicle 1.

Claims

[1] Method (10) for operating a driver assistance system (7) for traffic light visualization by means of a display (6) in a motor vehicle (1), comprising the following steps: - determining (11) a viewing direction of a driver (3) of the motor vehicle (1), - determining (12) an image of a corneal reflection on at least one eye of the driver (3), - Determining (13) traffic lights and their positions in the corneal endoscopy image and calculating associated traffic light positions in a vehicle environment (9) in front of the motor vehicle (1), - Evaluating (14) the determined viewing direction with regard to the calculated traffic light positions in the vehicle environment (9) and observing whether, how often and / or how long the respective traffic light is viewed by the driver (3), - evaluating (15) the traffic lights viewed by the driver (3) and identifying the traffic light that controls a lane on which the motor vehicle (1) is located, - tracking (16) the identified traffic light in the corneal endoscopy image, and - Activation (17) of the traffic light visualization in the display (6) as soon as the identified traffic light leaves the corneal endoscopy image. [2] Method (10) according to claim 1, characterized by , - that the traffic light which is observed most frequently and / or for the longest time by the driver (3) is identified as the traffic light which controls the lane on which the motor vehicle (1) is located. [3] Method (10) according to claim 1 or 2, characterized by , - that the evaluation (15) of the viewing direction with regard to the calculated traffic light positions in the vehicle environment (9), the observation of whether, how often and / or how long the respective traffic light is viewed by the driver (3), the evaluation of the traffic lights viewed by the driver (3) and the identification of the traffic light which controls the lane on which the motor vehicle (1) is located are continuously repeated. [4] Method (10) according to one of the preceding claims, characterized by , - that the determination (13) of the traffic lights and their positions in the corneal endoscopy image and the calculation of the associated traffic light positions in the vehicle environment (9) in front of the motor vehicle (1) are carried out by means of templates which are provided for a large number of different intersection situations controlled by traffic lights. [5] Method (10) according to one of the preceding claims, characterized by , - that the traffic light visualization in the display (6) is deactivated (18) as soon as the identified traffic light appears again in the corneal endoscopy image. [6] Method (10) according to one of the preceding claims, characterized by , - that the determination of the driver's (3) viewing direction is carried out by means of a driver observation camera (4), which is also used to determine the image of the corneal reflection on the respective eye of the driver (3). [7] Method (10) according to one of the preceding claims, characterized by , - that a video stream of the vehicle surroundings (9) in front of the motor vehicle (1) is generated by means of a front camera (5), which is shown in the respective display (6) when the traffic light visualization is activated. [8] Method (10) according to one of the preceding claims, characterized by , - that the identified traffic light is only tracked as long as this traffic light is located in the vehicle environment (9) in front of the vehicle (1). [9] Method (10) according to claim 8, characterized by , - that the traffic light visualization in the display (6) is deactivated as soon as the identified traffic light leaves the vehicle environment (9) in front of the vehicle (1). [10] Motor vehicle (1), - with a vehicle interior (2) in which a driver (3) can sit, - with a driver observation camera (4) for observing the driver (3) sitting in the vehicle interior (2), - with a front camera (5) for generating a video stream of a vehicle environment (9) in front of the motor vehicle (1), - with a display (6) arranged in the vehicle interior (2), - with a driver assistance system (7) which is coupled to the front camera (5) and to the display (6) and which is configured for traffic light visualization on the display (6), - with a control device (8) which is coupled to the driver observation camera (4) and to the driver assistance system (7) and which is configured to carry out the method (10) according to one of the preceding claims.

Images