Method for assisting a driver in steering the vehicle by activating a lane keeping assist system depending on the driver's direction of gaze, as well as an assistance system

Abstract

Method for assisting a driver (3) in driving a vehicle (1), in which a driver monitoring device (12) detects the direction of the driver's (3) gaze and transmits an activation signal to a control unit (9) of a lane keeping assistant (4) to activate the lane keeping assistant (4) if the driver monitoring device (12) detects that the driver (3) is looking away from a lane (5), wherein the activated lane keeping assistant (4) keeps the vehicle (1) in the lane (5) by means of lateral guidance, characterized in that the control unit (9), to which the activation signal is transmitted by means of the driver monitoring device (12), is part of a front camera device (6) of the lane keeping assistant (4), wherein the lane (5) in a surrounding area (8) of the vehicle (1) is detected by means of the front camera device (6).

Description

[0001] The present invention relates to a method for assisting a driver in driving a vehicle, in which the driver's gaze direction is detected by means of a driver monitoring device. Furthermore, the driver monitoring device transmits an activation signal to a control unit of a lane keeping assist system to activate the lane keeping assist system if it detects that the driver has taken their gaze away from the lane. With the lane keeping assist system activated, the vehicle is kept in its lane by lateral guidance. The present invention also relates to an assistance system for supporting a driver in driving a vehicle.

[0002] Various driver assistance systems are known from the state of the art to support the driver in operating a vehicle. For example, lane keeping assistants are known that issue a warning if the vehicle leaves its lane. Furthermore, lane keeping assistants are known that use steering intervention or lateral guidance to keep the vehicle within its lane. These lane keeping assistants, also known as active lane keeping systems (LKAS), typically feature a front camera that detects the lane in the vicinity of the vehicle or in front of it. Based on sensor signals from this front camera, the steering intervention or lateral guidance of the vehicle can then be controlled.It is known from the state of the art that these lane keeping assistants can be activated by a corresponding operating input.

[0003] Furthermore, it is known from the prior art that lane keeping assistants can be activated automatically, for example, if it is detected that the driver has taken their eyes off the lane or is currently inattentive. GB 2500690 A describes a driver monitoring system for this purpose, which can, for example, detect the position and / or movement of a driver. Depending on the position and / or movement of the vehicle, intervention in the vehicle's steering can be made.

[0004] Furthermore, DE 10 2015 122 603 A1 discloses a driver assistance system for a vehicle which includes an environmental sensor for detecting the vehicle's surroundings. A driver status monitor is also provided, which monitors the driver and detects any abnormal driving behavior. This abnormal behavior can be detected, for example, if the driver is looking away from the road. A lane keeping assist system can then be activated, for example, using a driver assistance system controller if the abnormal behavior is detected.

[0005] From DE 10 2016 220 549 A1, a method and a device for assisting a driver in deactivating a highly automated driving mode of a vehicle are known. Furthermore, from DE 10 2014 116 037 A1, a method for operating a driver assistance system of a motor vehicle is known.

[0006] The object of the present invention is to demonstrate a solution for how a functionality of a lane keeping assistant, which is activated depending on the driver's behavior, can be provided more efficiently.

[0007] This problem is solved according to the invention by a method and by an assistance system with the features according to the independent claims. Advantageous embodiments of the present invention are specified in the dependent claims.

[0008] A method according to the invention serves to assist a driver in driving a vehicle. Here, a driver monitoring device detects the driver's gaze direction. Furthermore, the driver monitoring device transmits an activation signal to a control unit of a lane keeping assist system to activate the lane keeping assist if it detects that the driver has taken their gaze away from the lane. The activated lane keeping assist system keeps the vehicle in its lane by means of lateral guidance. The control unit to which the activation signal is transmitted by the driver monitoring device is arranged in or on a front camera unit of the lane keeping assist system. The front camera unit captures the vehicle's surroundings.

[0009] The procedure can be carried out using a corresponding vehicle assistance system. During vehicle operation or while driving, the driver's gaze direction can be continuously recorded by the driver monitoring system. This driver monitoring system, which can also be referred to as a Driver Monitoring System (DMS), can include at least one camera. It is possible for the driver monitoring system to have two cameras or a stereo camera. The driver, and in particular the driver's head, can be recorded by the at least one camera of the driver monitoring system. Furthermore, the driver monitoring system can record the orientation of the driver's head and / or their gaze direction. The driver monitoring system can also include at least one distance sensor.Furthermore, the driver monitoring system can include a corresponding processing unit that uses images to determine whether the driver is looking at the road or away from it. From this, it can be deduced whether the driver is currently looking at the lane in which the vehicle is located. It can also be determined whether the driver is looking away from the road and, for example, towards the back seat.

[0010] If the system detects that the driver's eyes are diverted from the lane, the lane keeping assist can be automatically activated. For this purpose, the driver monitoring system transmits the activation signal to the lane keeping assist control unit. Upon receiving the activation signal, the lane keeping assist is activated, and the vehicle is kept within the lane by means of appropriate lateral guidance or steering interventions. The lane keeping assist can also be referred to as active lane keeping assist. The lane keeping assist also features a front camera system, which detects the lane in the vicinity of the vehicle and in the direction of travel ahead. Depending on the sensor signals from the front camera system, the lateral guidance can then be adjusted accordingly.For example, the lateral guidance can be adjusted so that the vehicle remains in the center of the lane. This allows the lane keeping assist to be activated automatically if the driver takes their eyes off the lane or roadway and / or is currently inattentive.

[0011] According to a key aspect of the present invention, the control unit to which the activation signal is transmitted via the driver monitoring device is part of a front camera system of the lane keeping assist system. This front camera system can include at least one camera with which the vehicle's surroundings, or the area in front of the vehicle in the direction of travel, can be captured. In particular, the camera of the front camera system can provide images that describe the lane. For this purpose, the images from the front camera system can be evaluated. Specifically, it can be provided that lane markings associated with the lane are recognized in the images. This can be done with the control unit of the front camera system or with a separate computing unit.The front camera system can, for example, have a corresponding housing in which at least one camera can be arranged, at least partially. The control unit can also be arranged in this housing. Alternatively, the control unit can be arranged on the at least one camera of the front camera system. According to the invention, the control unit of the front camera system is used to receive the activation signal from the driver monitoring system. This eliminates the need for an additional control unit compared to the prior art, which would otherwise only serve to detect the activation signal from the driver monitoring system and activate the lane keeping assist system based on the receipt of the activation signal. Therefore, the functionality of the lane keeping assist system, which is activated depending on the driver's behavior, can be provided more efficiently.

[0012] Preferably, the control unit receives sensor signals from the front camera system that describe the lane markings and uses these sensor signals to output control signals for the vehicle's lateral guidance. The sensor signals can be provided by the images captured by the at least one camera of the front camera system. Alternatively, the sensor signals can contain information determined from the evaluation of these images. In particular, the sensor signals describe the relative position of the lane markings to the at least one camera of the front camera system. These sensor signals can then be evaluated by the control unit, thus generating control signals for the lateral guidance. These control signals can, for example, be used to actuate a corresponding actuator that intervenes in the steering and thus changes the vehicle's steering angle.Overall, the control unit can receive and process both the activation signal and the control signals. This has the advantage that no additional control unit is required to evaluate the activation signal provided by the driver monitoring system.

[0013] In one embodiment, during the activation phase of the lane keeping assist system, steering angle changes are made for lateral guidance by means of the lane keeping assist system, depending on a predetermined initial steering angle change function. It has been found that an abrupt intervention by the steering angle assist system upon activation can contribute to driver uncertainty and, in the worst case, can endanger safety. When the lane keeping assist system is activated, the driver is usually not paying attention or is not currently focused on driving the vehicle. If, at this moment, the lane keeping assist system performs an abrupt change in the steering angle, for example, this can startle and thus unsettle the driver.It is also possible that the driver might make a steering maneuver as a result of the lane keeping assist system's steering angle change, potentially endangering the driver's safety and / or that of other road users. Therefore, an activation phase is provided for the transition from deactivated to activated lane keeping assist. This ensures a virtually seamless transition from deactivated to activated operation. During the activation phase, the steering angle change initiated by the lane keeping assist system can be limited, depending on the function. This limitation can be implemented, for example, as a threshold value. This means that the steering angle change during the activation phase does not exceed a predetermined limit.It can also be designed so that the function is pre-programmed in such a way that the steering angle change performed by the lane keeping assist system increases linearly. This way, driver uncertainty can be avoided.

[0014] In a further embodiment, the steering angle changes during the activation phase are carried out depending on a steering input currently provided by the driver. If the driver takes their eyes off the lane, for example, when looking towards the rear seats, they may unintentionally make a steering movement due to the rotation of their body. Particularly in this situation, an abrupt change in the steering angle caused by activation of the lane keeping assist system can have critical safety consequences. Therefore, it is specifically designed that the steering angle change made by the lane keeping assist system is determined based on the driver's current steering input. The driver can make this steering input, for example, by operating a steering wheel or steering handle.By taking into account the current steering input from the driver, the activation of the lane keeping assist system can be carried out safely.

[0015] Furthermore, it is advantageous if the lane keeping assist system is deactivated when the driver monitoring system detects that the driver is looking at the lane. As soon as the driver's eyes return to the lane or the road, the lane keeping assist system can be automatically deactivated. In this case, it can be assumed that the driver is once again focused on the traffic situation and is attentive, and thus able to steer the vehicle independently. By automatically activating and deactivating the lane keeping assist system based on the driver's gaze direction and attention level, the driver is provided with convenient support while driving. This also enhances safety during vehicle operation.

[0016] In another embodiment, during a deactivation phase for the lane keeping assist system, steering angle changes are performed depending on a predetermined second steering angle change function. As previously explained, a smooth transition between the deactivated and activated lane keeping assist systems can be provided. Furthermore, the transition between the activated and deactivated lane keeping assist systems can also be smooth. In this case, a deactivation phase is provided between the activated and deactivated lane keeping assist systems. During this deactivation phase, the steering angle changes performed by the steering angle assist system can also be predetermined depending on a function.As previously explained, the function can be predefined by setting a limit value for steering angle changes or a predetermined change. This minimizes the impact on the driver when the steering angle assist is deactivated.

[0017] In a further implementation, the driver receives visual, audible, and / or haptic feedback indicating that the lane keeping assist system is activated. This activation can be indicated to the driver, for example, via a suitable user interface (HMI - Human Machine Interface).

[0018] For example, the driver can be shown a corresponding symbol or text indicating that the lane keeping assist system has been activated. Haptic feedback, such as a vibration of the steering wheel, can also be provided. Furthermore, the interior lighting of the vehicle can be modified to alert the driver that the lane keeping assist system is active. This could involve, for example, activating ambient lighting. The ambient lighting can be controlled to either turn on or change its color. An audible alert can also be provided, such as adjusting the volume of the vehicle's sound system.It is specifically intended that this feedback or output be designed to minimize driver confusion or alarm. This ensures the driver is informed that the lane keeping assist system has been activated and also guarantees vehicle safety. Similarly, the system can provide the driver with appropriate feedback if the lane keeping assist system is deactivated.

[0019] In a further embodiment, the system detects that the driver has taken their eyes off the road if the driver monitoring device detects that the driver is looking towards a rear seat, at a vehicle control, at a mobile device, and / or at an area in the surroundings other than the road. For example, the driver might look towards the rear seats while driving. This is the case, for instance, if the driver is looking at and / or talking to passengers, especially children, in the rear seats. Furthermore, the driver might be looking at a vehicle control or display while the vehicle is in operation. The driver might also be looking at a mobile device, such as a smartphone. Finally, the driver might be looking at other areas in the surroundings, such as a traffic sign.Furthermore, the driver may be inattentive and / or at risk of falling asleep. These situations can be detected by the driver monitoring system, which then triggers the activation signal. This ensures the reliable operation of the assistance system.

[0020] An assistance system according to the invention serves to support a driver in driving a vehicle. The assistance system comprises a driver monitoring device for detecting the driver's gaze direction. Furthermore, the assistance system includes a lane keeping assistant for keeping the vehicle in its lane by lateral guidance. The lane keeping assistant includes a control unit. The driver monitoring device is configured to transmit an activation signal to the control unit to activate the lane keeping assistant if the driver monitoring device detects that the driver has taken their gaze away from the lane. The control unit is part of a front camera system of the lane keeping assistant, the front camera system being configured to detect the lane in the vicinity of the vehicle.

[0021] It is specifically provided that the front camera device includes a mounting element for attaching it behind the vehicle's windshield and / or to the vehicle's headliner. The front camera device can be positioned behind the windshield, allowing at least one camera to capture the vehicle's surroundings or the lane through the windshield. The front camera device can be attached to the windshield and / or the vehicle's headliner. Alternatively, a corresponding module can be arranged on the vehicle's headliner, to which the front camera device is attached.

[0022] A vehicle according to the invention comprises an assistance system according to the invention. It is particularly provided that the front camera unit of the lane keeping assist system or the assistance system is arranged behind the windshield of the vehicle and / or on the roof lining of the vehicle. The vehicle can be designed, in particular, as a passenger car. It can also be designed as a commercial vehicle.

[0023] The preferred embodiments and their advantages presented with reference to the method according to the invention apply accordingly to the assistance system according to the invention and to the vehicle according to the invention.

[0024] Further features of the invention are evident from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as those subsequently mentioned in the description of the figures and / or shown in the figures alone, are not only usable in the combinations specified but also in other combinations without departing from the scope of the invention. Thus, embodiments that are not explicitly shown and explained in the figures but can be derived and generated from the explained embodiments by separate combinations of features are also to be considered as encompassed and disclosed by the invention. Embodiments and combinations of features that do not exhibit all the features of an originally formulated independent claim are also to be considered disclosed.Furthermore, embodiments and combinations of features, in particular those set out above, are to be considered disclosed which go beyond or deviate from the combinations of features set out in the cross-references of the claims.

[0025] The invention will now be explained in more detail with reference to preferred embodiments and the accompanying drawings. These show: Fig. 1 a schematic representation of a vehicle which has an assistance system with a lane keeping assist; Fig. 2 a driver of the vehicle who is looking at a lane in the direction of travel ahead of the vehicle; and Fig. 3 of the drivers according to Fig. 2, who turns his gaze away from the lane.

[0026] In the figures, identical or functionally equivalent elements are given the same reference symbols.

[0027] Fig. Figure 1 shows a vehicle 1, in this case a passenger car, in a top view. The vehicle 1 includes an assistance system 2, which serves to support a driver 3 in driving the vehicle 1. The assistance system 2 includes a lane keeping assist system 4, by means of which the vehicle 1 can be kept in a lane 5. The lane keeping assist system 4 includes a front camera unit 6, which in turn has at least one camera 7. With this camera 7, the lane 5 in the area 8 in front of the vehicle 1 in the direction of travel can be monitored. The camera 7 can provide corresponding images that describe the area 8 or the lane 5. Furthermore, the front camera unit 6 includes a control unit 9, by means of which sensor data or images provided by the camera 7 can be evaluated.To identify lane 5, corresponding lane markings 10, which are assigned to lane 5, can be recognized in the images. The control unit 9 and the camera 7 can be arranged in a common housing. Alternatively, the control unit 9 can be arranged on or attached to the camera 7.

[0028] Furthermore, the lane keeping assist system 4 includes a fully schematically represented actuator 11, which can intervene in the steering of the vehicle 1. Actuator 11 can thus be used to adjust the steering angle of the vehicle 1 to influence lateral control. This actuator 11 can be controlled by the control unit 9. The control unit 9 also receives sensor data from the front camera system 6. Depending on these sensor signals, the actuator 11 can then be controlled by the control unit 9. For example, the steering can be adjusted so that the vehicle 1 is positioned in the center of lane 5.

[0029] Furthermore, the assistance system 2 includes a driver monitoring device 12, which can detect the direction of the driver's gaze and / or the orientation of the driver's head. The installation position of the driver monitoring device 12 shown here is for illustrative purposes only. The driver monitoring device 12, which is only shown schematically here, can have at least one camera with which images of the driver 3 or their head can be captured. These images can be evaluated by a processing unit of the driver monitoring device 12, and thus the direction of the driver's gaze can be determined. In particular, the driver monitoring device 12 can detect whether the driver 3 is looking at lane 5 or away from lane 5.

[0030] If it is detected that the driver 3 is taking their eyes off lane 5, an activation signal is transmitted from the driver monitoring device 12 to the control unit 9, and the lane keeping assist 4 is activated. In this case, the control unit 9 of the front camera system 6 is used to receive the activation signal from the driver monitoring device 12 and, depending on the activation signal, to activate the lane keeping assist 4 or the actuator 11. Therefore, a separate control unit for evaluating the activation signal from the driver monitoring device 12 is not required.

[0031] Fig. Figure 2 shows a driver 3 at the wheel of vehicle 1. During the journey of vehicle 1, the driver 3 is continuously monitored by the driver monitoring device 12. In this case, the driver monitoring device 12 is located in the area of ​​the headliner 14. For example, the driver monitoring device 12 can be located in a corresponding module, which is positioned on the inside of a windshield 13 in the area of ​​the headliner 14. The front camera device 6 can also be located in this module. The at least one camera of the front camera device 6 is positioned so that it can record the lane 5 in the surrounding area 8 through the windshield 13.

[0032] In the example of Fig. 2. The driver 3 focuses his gaze on lane 5 in the direction of travel in front of vehicle 1. Therefore, it is not necessary for the driver to be supported by the lane keeping assist system 4. In contrast, Fig. 3. Driver 3, who is looking towards the rear seats of vehicle 1 and thus away from lane 5. Driver 3 is currently not paying attention to lane 5 or the road traffic. If this situation is detected by the driver monitoring device 12, the activation signal is transmitted from the driver monitoring device 12 to the control unit 9, and the lane keeping assist 4 is automatically activated. If the lane keeping assist 4 is activated, this can be indicated to driver 3. This can be done by displaying a message on a screen, adjusting the interior lighting (e.g., ambient lighting), and / or adjusting the volume of the radio or sound system.

[0033] When the Lane Keeping Assist 4 is activated, a smooth transition is ensured between the deactivated and activated Lane Keeping Assist 4 states. During a designated activation phase, the steering angle change performed by the Lane Keeping Assist 4, or rather by the actuator 11, can be determined by a corresponding function. This means, in particular, that steering angle changes performed by the actuator 11 remain below a defined threshold. This prevents the driver 3 from being startled by abrupt steering angle changes performed by the Lane Keeping Assist 4.

[0034] As soon as the driver monitoring device 12 detects that the driver 3 has turned his attention back to lane 5, the lane keeping assist 4, or rather its function, can be automatically deactivated. Here, too, a smooth transition from the activated state to the deactivated state of the lane keeping assist 4 is provided, so that the driver 3 is affected as little as possible by the transition or does not perceive it at all.

Claims

[1] Method for assisting a driver (3) in driving a vehicle (1) in which a driver monitoring device (12) detects the direction of the driver's (3) gaze and transmits an activation signal to a control unit (9) of a lane keeping assist system (4) to activate the lane keeping assist system (4) if the driver monitoring device (12) detects that the driver (3) is taking his gaze away from a lane (5), wherein the activated lane keeping assist system (4) keeps the vehicle (1) in the lane (5) by means of lateral guidance, characterized by , that the control unit (9) to which the activation signal is transmitted by means of the driver monitoring device (12) is part of a front camera device (6) of the lane keeping assist system (4), wherein the lane (5) in a surrounding area (8) of the vehicle (1) is detected by means of the front camera device (6). [2] Method according to claim 1, characterized by, that sensor signals describing the lane (5) are received by means of the control unit (9) from the front camera device (6) and control signals for the lateral guidance of the vehicle (1) are output on the basis of the sensor signals. [3] Method according to claim 1 or 2, characterized by , that during an activation phase to activate the lane keeping assist (4) the lane keeping assist (4) makes steering angle changes for lateral guidance depending on a predetermined first steering angle change function. [4] Method according to claim 3, characterized by , that the steering angle changes during the activation phase are carried out depending on a steering input currently specified by the driver (3). [5] Method according to any one of the preceding claims, characterized by, that the lane keeping assist (4) is deactivated if the driver monitoring device (12) detects that the driver (3) is looking at the lane (5). [6] Method according to claim 5, characterized by , that during a deactivation phase to deactivate the lane keeping assist (4) steering angle changes are carried out depending on a predetermined second steering angle change function. [7] Method according to any one of the preceding claims, characterized by , that the driver (3) of the vehicle (1) receives visual, acoustic and / or haptic feedback if the lane keeping assist (4) is activated. [8] Method according to any one of the preceding claims, characterized by, that it is recognized that the driver (3) is taking his gaze away from the lane (5) if it is detected by means of the driver monitoring device (12) that the driver (3) is looking towards a rear seat of the vehicle (1), towards a control element of the vehicle (1), towards a mobile device of the driver (3) and / or towards an area in the surroundings (3) other than the lane (5). [9] Method according to any one of the preceding claims, characterized by , that the front camera device (6) has a housing in which at least one camera (7) of the front camera device (6) is arranged, and wherein the control unit (9) is also arranged in this, in particular common, housing. [10] Method according to any one of the preceding claims, characterized by that the front camera device (6) has at least one camera (7) on which the control unit (7) is arranged. [11] Assistance system (2) for assisting a driver (3) in driving a vehicle (1) comprising a driver monitoring device (12) for detecting the direction of the driver's (3) gaze, a lane keeping assist (4) for keeping the vehicle (1) in a lane (5) by lateral guidance, wherein the lane keeping assist (4) includes a control unit (9), and wherein the driver monitoring device (12) is configured to transmit an activation signal to the control unit (9) to activate the lane keeping assist (4) if the driver monitoring device (12) detects that the driver (3) is taking his / her gaze away from the lane (5), characterized by , that the control unit (9) is part of a front camera device (6) of the lane keeping assist system (4), wherein the front camera device (6) is designed to detect the lane (5) in a surrounding area (8) of the vehicle (1). [12] Assistance system (2) according to claim 11, characterized by, that the front camera device (6) has a fastening element for attaching the front camera device (6) behind a windscreen (13) of the vehicle (1) and / or to a roof liner (14) of the vehicle (1). [13] Assistance system (2) according to claim 11 or 12, characterized by , that the front camera device (6) has a housing in which at least one camera (7) of the front camera device (6) is arranged, and wherein the control unit (9) is also arranged in this, in particular common, housing. [14] Assistance system (2) according to any one of the preceding claims 11 to 13, characterized by that the front camera device (6) has at least one camera (7) on which the control unit (7) is arranged.

Images