Method and assistance system for behavior-dependent warning of a driver of hazards and correspondingly equipped motor vehicle

Abstract

Method for warning a driver (8) of a motor vehicle (4) of dangers in its surroundings, wherein during the travel of the motor vehicle (4) - the surroundings of the motor vehicle (4) are continuously monitored for objects (5, 6, 7) located there, - the driver’s behaviour (8) is continuously recorded, - based on the recorded behaviour of the driver (8), a driving manoeuvre (9) likely intended by the driver is determined, - those objects (5, 6, 7) detected in the surroundings of the motor vehicle (4) which are relevant for this driving manoeuvre (9) are determined, - a warning (21) is issued only on these specific relevant objects (7, 19), and a) driver behavior data indicating the behavior of the driver (8) of the motor vehicle (4) and corresponding driving maneuver data indicating driving maneuvers (9) actually performed by the driver (8) according to recorded driver behavior data are provided as training data for a machine learning device by an assistance system (13) for the motor vehicle (4) set up to carry out the method, and an output of the machine learning device trained therewith is used to determine or indicate the respectively expected intended driving maneuver (9), and / or b) the assistance system (13) for the motor vehicle (4) set up to carry out the method records routes travelled by the driver (8) over several journeys of the motor vehicle (4) as part of the driver behaviour data and a resulting movement profile of the driver (8) is taken into account to determine the respective likely intended driving manoeuvre (9).

Description

[0001] The present invention relates to a method and an assistance system for warning a driver of a motor vehicle of hazards. The invention further relates to a correspondingly configured motor vehicle.

[0002] Dangerous situations and accidents repeatedly occur in road traffic when, for example, drivers fail to see other road users in their surroundings. Although mirrors are usually provided in motor vehicles to help the driver get an overview of their surroundings, experience has shown that this is not always sufficient, for example in situations with a large number of other road users in the vicinity and / or high relative speeds between the vehicle and other road users. For example, a quick glance in the mirror may not always correctly identify all of the road users visible there, and a brief glance at the mirror may not reveal relative speeds. There is therefore a need for further improvements in this area.

[0003] As one approach, DE 10 2009 002 277 A1 describes a driver assistance system for warning the driver of a vehicle about obstacles in the vehicle's surroundings. An evaluation unit generates a reference value based on sensor data and compares it with a subsequent instantaneous value for obstacle detection, also generated based on the sensor data. A warning is then issued if the instantaneous value deviates from the reference value. The reference value is also generated before the vehicle starts moving if a specific driver behavior is detected.

[0004] DE 10 2011 012 793 A1 describes a driver assistance method for displaying and / or autonomously or semi-autonomously controlling a collision-avoiding or collision-mitigating driving maneuver of a motor vehicle. This method determines an evasive trajectory based on the current surrounding situation and a trajectory prediction for the motor vehicle and for a detected object in the surrounding area, taking driver characteristics into account. The driving maneuver based on the determined evasive trajectory is then displayed and / or controlled.

[0005] DE 11 2019 004 597 T5 describes an information processing device comprising a data processing unit that determines a level of alertness corresponding to the level of consciousness of a driver of a vehicle. The data processing unit obtains observation information about the driver and determines the driver's level of alertness based on the obtained observation information.

[0006] DE 11 2014 001 436 T5 describes a method for controlling vehicle systems in a motor vehicle. The method comprises obtaining information from a first vehicle system, determining a level of drowsiness, detecting a hazard, modifying the control of the first vehicle system based on at least the level of drowsiness, and selecting a second vehicle system. The second vehicle system differs from the first vehicle system. The method further comprises modifying the control of the second vehicle system based on at least the level of drowsiness.

[0007] DE 10 2012 002 149 B3 describes a method for visualizing the surroundings of a motor vehicle. The method involves recording the surroundings using a camera and displaying the camera image on a screen. Furthermore, different colored frames are used for vehicles with different relative speeds, with the color of the frame indicating whether the detected vehicle is one with a critical relative speed or not. This is intended to support the driver by displaying processed objects in the vehicle's surroundings.

[0008] As another approach, DE 20 2006 010 960 U1 describes a video surveillance system for use in motor vehicles. Several cameras are mounted around a vehicle to record the surroundings. A digital image processing unit, which contains an image display unit and a unit for highlighting moving objects, serves as an intermediate link between the cameras and a display unit within the vehicle. The image display unit combines the raw images from the individual cameras into a flat image for display on the display unit. The unit for highlighting moving objects places them in a frame that tracks movement within the cameras' detection range. This is intended to increase security.

[0009] The object of the present invention is to enable improved safety in traffic.

[0010] This object is achieved by the subject matter of the independent claims. Further possible embodiments of the invention are disclosed in the subclaims, the description, and the figures. Features, advantages, and possible embodiments presented in the description for one of the subject matter of the independent claims are to be regarded at least analogously as features, advantages, and possible embodiments of the respective subject matter of the other independent claims, as well as any possible combination of the subject matter of the independent claims, optionally in conjunction with one or more of the subclaims.

[0011] The method according to the invention serves to warn a driver of a motor vehicle of hazards in their surroundings. The method can be carried out automatically while the motor vehicle is moving, for example using an appropriately configured assistance system of the motor vehicle. In the method according to the invention, the surroundings of the motor vehicle are continuously monitored for objects located there. Such objects can in particular be other road users or static obstacles such as bollards, fences, walls, lampposts or the like. The respective surroundings or surrounding traffic events can therefore be observed here by means of an appropriate environmental sensor system of the motor vehicle, such as at least one camera and / or a radar device and / or a lidar device.Using appropriate sensor data, objects in the environment can be automatically detected, for example by means of appropriate image or data processing, particularly based on machine learning. Likewise, as part of the monitoring of the environment, Car2Car data can be collected from the environment or from other road users or vehicles in the vicinity of the motor vehicle, and other vehicles or road users in the vicinity can be detected based on this. Such Car2Car data can, for example, indicate the types, positions, speeds, directions of travel, and / or planned trajectories of the other vehicles or road users. For example, an approaching vehicle can communicate via a Car2Car connection that it is approaching the motor vehicle at a certain speed and / or that it is likely to enter its vicinity.This allows relevant objects to be identified, if necessary, even if the vehicle itself lacks its own environmental sensors or if these sensors are impaired. Likewise, the position of the vehicle can be continuously recorded or tracked, for example. Based on this, map data, particularly high-resolution or highly detailed map data, can then be evaluated. For example, static objects in the vehicle's surroundings can be identified based on the map data. Likewise, a comparison or plausibility check can be performed between the sensor data or objects detected therein and the map data.

[0012] Furthermore, the method according to the invention continuously records and monitors the driver's behavior. For this purpose, the driver can be observed, for example, using an interior camera of the motor vehicle. As part of the driver's behavior, gestures, gaze directions, gaze changes, head positions, hand positions, foot positions, operating actions, verbal utterances, and / or the like can be recorded. Likewise, the driver's mental and / or physical state can be recorded and / or monitored. For example, a state of alertness or fatigue can be determined, and / or pupil movements, for example in terms of extent, pattern, frequency, and speed, can be recorded.

[0013] Furthermore, in the method according to the invention, a driving maneuver that is likely intended by the driver, i.e., a driving maneuver that is likely planned or imminent, is determined or predicted based on the detected driver behavior. For this purpose, for example, an appropriately trained machine learning device can be used, which is trained to predict the driver's likely subsequent or intended driving maneuvers based on driver behavior data that indicate, depict, or characterize the driver's behavior. Likewise, for example, predefined criteria and / or a predefined assignment table can be evaluated in order to assign detected driver behavior(s) to a driving maneuver that is likely intended by the driver.If, for example, a driver's behavior in right-hand traffic is detected as a glance over his shoulder to the left and a brief movement of the steering wheel to the left to steer the vehicle to the left edge of the lane currently being used, this can be inferred from a likely intention to change lanes into the adjacent lane on the left. For example, if the driver glances more than just past a traffic sign indicating an upcoming exit and subsequently or concomitantly reduces speed, and if the driver positions his hand within reach of a control element for a turn indicator, it can be inferred from a likely imminent driving maneuver to leave the road currently being used via the upcoming exit.Likewise, a variety of other driver behaviors or behavior patterns can indicate a variety of other driving maneuvers.

[0014] Likewise, map and / or environmental data for the respective current surroundings and those ahead in the direction of travel of the motor vehicle can be taken into account or evaluated. On the basis of this, it can be determined, for example, which driving maneuvers of the motor vehicle are actually possible, sensible or plausible, or whether the determined, probably intended driving maneuver is possible, sensible or plausible in the respective surroundings. For example, it can be determined that the motor vehicle is driving in the right-hand traffic system in the right of two lanes intended for the same direction of travel, and that the left lane ends immediately ahead or merges with the right lane. In such a situation, it can then be determined or decided, for example, that the driver is unlikely to plan an overtaking maneuver.Corresponding glances by the driver in the left-hand side mirror, which might otherwise be typical of an impending overtaking maneuver, can then be interpreted differently, for example, or the intended driving maneuver initially identified on the basis of these glances can then be rejected based on the map and / or environmental data. As part of the map or environmental data, for example, lane layouts, speed limits, overtaking bans, the positions of restricted areas, and / or similar can be recorded and taken into account or evaluated. Likewise, the speed of one or more other road users orMotor vehicles in the vicinity of the motor vehicle, in particular in or only in the area ahead of the motor vehicle in the direction of travel, relative to the motor vehicle and / or the operating state of at least one driver assistance system of the motor vehicle configured for longitudinal guidance or longitudinal control of the motor vehicle are determined and taken into account in order to determine or verify the plausibility of the likely intended driving maneuver. Based on this, it can be determined, for example, that the motor vehicle is currently being controlled with active adaptive cruise control in order to follow a vehicle traveling ahead at the same speed.In this case, for example, a manual overtaking maneuver can be rejected as implausible, or it can be provided that, in this case, detection of a likely intended driving maneuver, such as an overtaking maneuver that requires or includes overriding the active automated cruise control, is determined only based on additional or different behaviors than the likely intended one. This can apply, for example, in comparison to a situation - for example, an otherwise identical situation - in which the speed of the motor vehicle is not adapted to the speed of one or more other vehicles in the surrounding area, particularly those ahead, or is not automatically controlled.

[0015] In a first variant of the present invention, an assistance system for the motor vehicle configured to carry out the method provides driver behavior data indicating the behavior of the driver of the motor vehicle, and corresponding driving maneuver data indicating driving maneuvers actually performed by the driver based on acquired driver behavior data, as training data for a machine learning device. An output from the machine learning device trained therewith is used to determine or indicate the respectively anticipated intended driving maneuver. In principle, the use of the machine learning device proposed here for processing the driver behavior data can enable a particularly accurate and reliable determination of the respectively anticipated intended driving maneuver.In addition, the present invention can be implemented comparatively simply and with little effort, for example in comparison to a manual implementation of a recognition of the driver's behavior and a decision algorithm for determining the likely intended driving maneuver therefrom.

[0016] Additionally or alternatively, in a second variant of the present invention, the assistance system for the motor vehicle configured to execute the method records routes traveled by the driver over several journeys of the motor vehicle as part of the driver behavior data, and a resulting driver movement profile is then taken into account to determine the respective likely intended driving maneuver. This enables a particularly accurate and reliable determination of the actually intended driving maneuver.

[0017] Furthermore, the method according to the invention determines those objects detected in the respective current surroundings of the motor vehicle which are actually or presumably relevant for the determined driving maneuver presumably intended by the driver. Relevant objects can be objects which must be observed or taken into account for the safe execution of the driving maneuver, or which are or could become actually or potentially dangerous when the driving maneuver is carried out, or which represent an obstacle, or with which there is a risk of collision when the driving maneuver is carried out, or the like. Objects which must be observed or taken into account can, for example, be objects which are or would be at least temporarily within a predetermined maximum distance from the motor vehicle when the driving maneuver is carried out.To determine the relevant objects, for example, movement data of the motor vehicle and the detected objects can be evaluated or extrapolated, or movements or relative movements of the motor vehicle and the detected objects can be modeled or simulated, or the like.

[0018] Furthermore, in the method according to the invention - at least if at least one relevant object has been detected - a warning is issued only for the specific relevant objects. This means that the driver's attention can be directed or focused precisely on these objects. For this purpose, for example, in a display device of the motor vehicle only the relevant objects can be highlighted or marked compared to their surroundings and / or other detected objects that were not classified as relevant for the presumably intended driving maneuver. The type or design of the warning can be adapted to the previously detected or determined mental and / or physiological state of the driver. For example, the warning can be selected or optimized to particularly effectively attract the driver's attention and / or avoid startling the driver.For this purpose, for example, a corresponding specification, for example in the form of a table or a characteristic map, can be taken into account, or a predefined model or a correspondingly trained machine learning device can be used to determine the optimal type and / or design of the warning to be used in each case. The driver's current direction of view can also be taken into account, for example. This means that the warning or part of it can, for example, only be output or at least also on one or more of the display devices of the motor vehicle that are currently in the driver's field of vision, for example at least or only on the central interior mirror or a side mirror or an instrument cluster or a display in the center console or a HUD. If the driver uses AR, VR or XR glasses or a corresponding headset, at least one virtual spatial area can be used to output the marking orof the warning. The output or presentation of the warning can also follow the driver's field of vision. If the driver turns his gaze from a first display device or a first direction of vision to a second display device or in a second direction of vision, the warning can be output accordingly first, in particular only, on the first display device or in the first direction of vision and then, in particular only, on the second display device or in the second direction of vision. This can, if necessary, improve the assurance that the driver sees the warning, and distractions for the driver due to changes in the presentation on a display device that is only in the driver's peripheral field of vision can be avoided.

[0019] The present invention can enable the driver to recognize relevant objects or potential hazards particularly quickly and easily and with particularly low cognitive load. Likewise, the warning can enable the driver to detect the presence and relevance of objects that, without the warning, they would not be able to correctly recognize or identify or assess in terms of their position and / or speed relative to the motor vehicle. This enables the driver to react accordingly and in a timely and targeted manner to avoid dangerous situations. This relieves the driver of the burden of having to completely assess the relevance of various recognizable objects themselves. Furthermore, the driver does not have to keep their gaze fixed on the mirror or to the side or rear for extended periods in order to reliably detect the relative movements of other road users.This allows the driver, for example, to monitor the traffic ahead in the direction of travel or the wider surroundings of the vehicle more reliably, i.e., with shorter pauses. The present invention can thus contribute to increasing safety and relieving the driver's workload, thus correspondingly increasing comfort. This can be achieved, in particular, without the driver having to fundamentally change their behavior compared to using conventional motor vehicles. The driver therefore does not have to, for example, consciously behave in a certain predetermined manner, nor explicitly state or communicate their intended driving maneuvers. This allows the present invention to be used particularly simply and reliably.

[0020] In particular, it can be provided here that the warning is only issued once the impending driving maneuver, i.e. the corresponding driver intention, is detected, and at the latest until the driving maneuver has been successfully completed or until it has been determined that the driving maneuver is not likely to be carried out. In other words, for example, not all detected objects or all detected objects within a predetermined distance from the motor vehicle and / or at a predetermined relative speed to the motor vehicle are permanently highlighted or marked. Likewise, if, for example, the driver randomly or routinely glances in the mirror or at the surroundings without any further indications of a likely intended driving maneuver, the detected objects may not be highlighted or marked accordingly, and no other warning may be issued.This can prevent the driver from being distracted or overwhelmed by a multitude of markings, highlights, or warnings. This can lead to, or contribute to, the driver recognizing a warning issued according to this method more quickly, reliably, and consciously, and giving it greater importance. This effect can also improve road safety compared to other approaches.

[0021] Likewise, the driver's behavior or reaction to an issued warning can be recorded automatically. In particular, it is also possible to record or determine whether the driver has carried out the identified, presumably intended, driving maneuver, or which driving maneuver the driver then actually carries out. It is also possible to determine whether a critical situation, a near-accident, a subsequent evasive maneuver, emergency braking, or the like occurs, or whether, for example, all specified speed limits and safety distances were observed. Based on this, it can then be automatically determined, in particular using machine learning, in which situations a warning is appropriate and / or which type, design, and / or output position of warnings is appropriate. This can then be taken into account for future warnings.determine future issuance of warnings.

[0022] The present invention also relates to an assistance system for a motor vehicle. The assistance system according to the invention has an input interface for acquiring environmental data that depicts or characterizes a respective environment or indicates objects located therein, and driver behavior data that indicates, for example, depicts or characterizes, the behavior of a driver of the motor vehicle. Furthermore, the assistance system has a data processing device for processing the acquired data and for determining a respective likely intended driving maneuver and relevant objects in the environment based on the acquired data, and for generating a respective corresponding control signal for generating a warning message only for precisely the relevant objects.For this purpose, the data processing device can, for example, comprise a processing device, such as a microchip, microprocessor, microcontroller, or the like, and a computer-readable data memory coupled thereto. Likewise, the data processing device can be, comprise, or implement a machine learning device, such as an artificial neural network. Furthermore, the assistance system according to the invention has an output interface for outputting the control signals. The input interface and the output interface can be different from one another, combined with one another, or integrated, such as being configured as a single bidirectional interface. The input interface and the output interface can each be implemented entirely or partially in hardware and / or software.The assistance system according to the invention is configured to carry out the method according to the invention, in particular automatically. For this purpose, a corresponding operating or computer program can be used, for example, which codes or implements the method steps, measures or sequences or corresponding control instructions described in connection with the method according to the invention or also subsequently in connection with the assistance system according to the invention and / or motor vehicle. This operating or computer program can, for example, be stored in the aforementioned data memory and be executable by means of the aforementioned processor device in order to carry out the corresponding method or to bring about its execution. The assistance system according to the invention can, in particular, be the assistance system mentioned in connection with the method according to the invention.For example, the assistance system according to the invention can be designed as a control unit for a motor vehicle or as a part or module of a more comprehensive vehicle system.

[0023] In a first variant of the present invention, the assistance system is configured to provide the driver behavior data for the driver and corresponding driving maneuver data, which indicate driving maneuvers actually performed by the driver based on recorded driver behavior data or driver behaviors, as training data for a machine learning device. Furthermore, the assistance system is configured to use an output of the corresponding machine learning device trained therewith to determine or indicate the likely intended driving maneuver. The assistance system can itself comprise the machine learning device.Likewise, the machine learning device can be a device different from the assistance system itself, in particular one external to the vehicle, for example an external server device, such as a cloud server or backend, a central data center, or the like. The assistance system can then be configured to send the driver behavior data and the driving maneuver data to the external machine learning device—for example, via a wireless data connection, such as a Wi-Fi or cellular connection.

[0024] The assistance system can be configured to send driver behavior data recorded during the journey to the external machine learning device and to collect data sent by the device in response, indicating a likely intended driving maneuver of the driver, determined or predicted by the machine learning device. Based on this, the assistance system can then determine the relevant objects in the environment.

[0025] The use of a local machine learning device, i.e., one provided as part of the assistance system or the motor vehicle, can enable particularly fast and low-latency processing of driver behavior data for applying or implementing the method according to the invention, without relying on an existing data connection. Furthermore, the machine learning device can then easily be individually trained for each driver, i.e., it can learn or have learned the individual driver-specific behavior patterns or behaviors of each driver with particular precision.

[0026] The use of an external, particularly vehicle-external, machine learning facility to process driver behavior data can, however, more easily enable the use of more or more powerful computing resources and / or the use of data from a large number of drivers and thus a particularly robust and reliable determination or prediction of the intended driving maneuver in each case.

[0027] In principle, the proposed use of the machine learning device for processing driver behavior data can enable a particularly accurate and reliable determination of the likely intended driving maneuver. Furthermore, the present invention can be implemented comparatively simply and with little effort, for example, compared to a manual implementation of driver behavior detection and a decision algorithm for determining the likely intended driving maneuver from it.

[0028] Additionally or alternatively, in a second variant of the present invention, the assistance system is configured to record routes traveled by the driver over multiple journeys of the motor vehicle as part of the driver behavior data and to use or take into account a resulting long-term movement pattern or movement profile of the driver, learned in particular by means of machine learning, to determine the respective likely intended driving maneuver. Thus, for example, a route plan or timetable, i.e. a driver routine, can be determined here. From this, a likely upcoming driving maneuver, for example for a lane change or turning or the like, can then be derived or predicted, even when navigation guidance is not active.This is based on the knowledge that a large number of journeys are routine journeys that are regularly repeated in the same way and / or that the driver's journeys or behavior can follow a certain recurring plan or pattern. For example, it can be determined that the driver is currently on a journey or is following a route that they repeat regularly. This can take into account not only position or movement data, but also time data, such as the current time of day and / or the current day of the week or similar. For example, it can be recognized that the driver is currently on their morning commute from their home to their regular place of work. Based on previous such journeys, it can then be known or learned which driving maneuvers the driver usually performs on this stretch or route.For example, it can be determined or learned that the driver only performs overtaking maneuvers at certain points on a regularly traveled route. Based on this, a determination, prediction, or plausibility check of the currently anticipated or intended driving maneuver can be performed.

[0029] The second variant of the present invention can be combined with the other variant or other embodiments. For example, the determined or learned movement profile can be used as one of several inputs for determining the likely intended driving maneuver and / or, for example, for validating or verifying the plausibility of the likely intended driving maneuver. For example, both data on the movement profile or a resulting likely intended driving maneuver as well as the current or actual body movements and / or operating actions of the driver during the current journey or a resulting likely intended driving maneuver can be combined or compared with one another. This enables a particularly precise and reliable determination of the actually intended driving maneuver.

[0030] In one possible embodiment of the present invention, the assistance system is designed to record, as part of the driver behavior data, data from which current movements and / or operating actions of the driver during the current journey can be derived. In other words, movements and / or operating actions recorded during the current journey, in particular during the current journey only within a sliding time window of a predetermined length extending from the current point in time into the past, can be used to determine the likely intended driving maneuver or maneuvers. Corresponding current movements of the driver can be, for example, head movements, glances to the side or over the shoulder or into a rear-view mirror, positions or changes in position of hands and / or feet, turning or straightening of the torso orThe driver's upper body and / or the like. Such data can be or include, for example, camera data from an interior camera or driver observation camera of the motor vehicle and / or data or signals from operating elements or a user interface or a control unit of the motor vehicle coupled thereto. The embodiment of the present invention proposed here can achieve or ensure that, in every situation, at least particularly relevant driver behavior data is used or taken into account to determine the respective likely intended driving maneuver.For example, such momentary movements or operating actions of the driver can indicate an intended driving maneuver that contradicts the long-term average behavior of the driver and / or unusual behaviors or behavior patterns of the driver can be attributed to a presumably intended driving maneuver in a longer-term comparison.

[0031] In a further possible embodiment of the present invention, the assistance system is configured to use the environmental data to predict the probable future movement behavior of detected objects in the environment of the motor vehicle relative to the latter and to identify the relevant objects based on the respective prediction. The future movement behavior can be predicted, for example, using a predefined movement or traffic model and / or using a correspondingly trained machine learning device. For this purpose, positions, movement states, directions of movement, speeds, accelerations, activities or states of direction indicators and / or brake lights and / or the like of other vehicles derived from or specified in the environmental data, as well as, for example, hand signals from cyclists or the like, can be taken into account.In particular, the future movement behavior of the detected objects can be predicted through a simple extrapolation of the current movement state, or even beyond. For example, anticipated driving maneuvers, such as lane changes or turning maneuvers by other vehicles detected as objects in the surrounding area, can be predicted and taken into account to determine the relevant objects. This enables particularly accurate and reliable determination of the objects relevant to the driver's intended driving maneuver.

[0032] If there are several objects in the vicinity of the motor vehicle, their likely future movement behavior can be predicted jointly or in combination, for example by means of a corresponding group or swarm simulation. Likewise, the likely future movement behavior of one or more of the objects detected in the vicinity of the motor vehicle can be predicted individually or in subgroups. For example, the objects can be prioritized according to their relative speed to the motor vehicle and / or the lane they are traveling in. For example, objects that are moving at a relative speed to the motor vehicle that is above a predetermined threshold and / or are moving in a lane targeted by the driver or the motor vehicle in accordance with the likely intended driving maneuver can be examined or classified as a matter of priority.The expected future movement behavior of such objects can be individually predicted. Since this can be less computationally intensive, the driver can be warned particularly quickly or at a particularly early stage about such potentially relevant objects. Likewise, such objects, if classified or identified as relevant for the intended driving maneuver, can be highlighted above other relevant objects, i.e., marked or displayed differently, particularly more conspicuously. Thus, different objects can be marked or highlighted differently depending on their relevance or—at least their potential—danger.

[0033] In a further possible embodiment of the present invention, the assistance system is designed to generate the warning message by controlling a mirror display of the motor vehicle to display a respective marking on the relevant objects visible in the mirror display as a warning message. In the present sense, a mirror display can be, for example, a mirror designed as a display, i.e. as an actively controllable display device, in particular an interior rear-view mirror, or a display or display device for a digital side mirror. In such a display for a digital side mirror, a camera image or video stream from an environmental observation camera of the motor vehicle arranged instead of a conventional side mirror can be shown. On the respective mirror display, the marking or warning message can be displayed as a marking orA warning message such as a frame or a circle or a highlighted outline or the like may be displayed for a relevant object.

[0034] Likewise, the direction of travel and / or the speed or relative speed of the respective object relative to the motor vehicle can also be displayed or indicated in addition to or as part of the marking. A relevant object can, for example, be outlined or highlighted in color, with the criticality or relevance, i.e., the degree of relevance of the respective object, being indicated in color. For example, a particularly relevant object, which, for example, represents a direct collision risk, can be marked red, while a relevant but less critical or less dangerous object, with which there is only a lower probability of collision, can be marked yellow or orange.

[0035] Additionally or alternatively, the assistance system can be configured to briefly display or color the entire image displayed in the respective mirror display - or in general in the respective display device - in a predetermined signal color when at least one relevant object is detected and / or to frame the entire displayed image with a frame, in particular in a predetermined signal color. Likewise, instead of a single color, a predetermined color sequence can be used for marking and / or displaying, coloring, or framing the entire image. The display of the marking of relevant objects in a mirror display proposed here, which can be used by the driver anyway to observe the surroundings of the motor vehicle, can enable the driver to recognize the relevant objects particularly easily, quickly, and with little distraction.

[0036] In a further possible embodiment of the present invention, the assistance system is configured to automatically increase or magnify the intensity or conspicuity of the warning compared to a respective initial display if or as soon as a predefined criterion is met. This can be the case, in particular, if no corresponding change in the driver's behavior or the likely intended driving maneuver is detected within a predefined period of time following the warning issued in the initial display. The intensity or conspicuity of the warning can also be increased in several steps or stages according to a predefined escalation cascade. For example, to increase the intensity or conspicuity, the brightness and / or color and / or size and / or line type or dashed lines and / or a temporal display pattern of the warning can be adjusted.The latter can mean, for example, that the warning is initially displayed constantly or continuously over time and then flashes to increase its intensity or conspicuity. The embodiment of the present invention proposed here can increase the probability that the driver will perceive the warning and react accordingly. This can contribute to further improving traffic safety using the present invention.

[0037] In general, the type and nature of the marking or warning, for example with regard to colour and / or line thickness and / or line pattern and / or text, can be individually adjusted or designed based on at least one predefined property of the respective object. This can, for example, be predefined or learned automatically, for example based on an effectiveness that is subsequently recorded or assessed - automatically and / or by the driver. The marking or warning for an object can also be dynamically adapted, i.e. changed, with changing relevance or with a changing value of at least one predefined property. For example, the conspicuousness or intensity of the marking orof the warning message is automatically increased as the distance of the object from the motor vehicle decreases and / or as the approach speed between the object and the motor vehicle increases.

[0038] In a further possible embodiment of the present invention, the assistance system is designed to automatically control or activate at least one further device, for example of the motor vehicle or of the assistance system, to output a further warning which addresses a different sense of the driver, if or as soon as a predefined criterion is met. This can be the case in particular if, following the initial warning, no corresponding change in the driver's behavior or the presumably intended driving maneuver is detected within a predefined period of time. The output of one or more such further warnings proposed here can, for example, take place according to a predefined escalation cascade. If the original orIf the initial warning is, for example, an optical or visual indication, for example on the mirror display mentioned elsewhere, an audio indication or a haptic indication can then be issued as a further warning that appeals to a different sense than the driver's sense of sight. An audio indication can, for example, be or include a signal tone or a voice output indicating at least one detected relevant object or a corresponding hazard, or the like. A haptic indication can, for example, be a mechanical impulse or a vibration. Such a haptic indication can be generated by means of a corresponding actuator on or in a component of the motor vehicle that is touched by the driver or is in contact with the driver. Such a component can, for example, be a steering wheel or a seat or an armrest of the motor vehicle.The haptic cue can be generated at a location or on a side of the respective component or driver that corresponds to the position of the detected relevant object relative to the motor vehicle.

[0039] As a criterion for issuing the additional warning, the reaching or exceeding of a predefined threshold value for the critical quality or relevance or danger, or a collision probability when executing the anticipated intended driving maneuver, can also be specified, used, or evaluated. In an exemplary application of the embodiment of the present invention proposed here, the assistance system can, for example, cause a vibration of the steering wheel and / or the driver's seat of the motor vehicle and / or the output of a signal tone or a voice output. Likewise, the assistance system can control a ventilation system of the motor vehicle to issue the additional warning, i.e., to alert the driver to the current situation.This allows the driver, for example, to be exposed to a blast of air, particularly from a direction corresponding to the position or approach direction of the respective detected relevant object relative to the motor vehicle. The proposed approach, particularly escalating, activating various senses of the driver—i.e., the use of various devices for issuing warnings—can increase the likelihood that the driver will recognize the actual or potential danger before actually executing the intended driving maneuver.

[0040] The present invention also relates to a motor vehicle having an environmental sensor system for detecting objects in the surroundings of the motor vehicle, a detection device, in particular a driver observation device, for detecting the behavior of the driver of the motor vehicle, i.e., corresponding driver behavior data, the assistance system according to the invention, and at least one output device coupled thereto for outputting warnings to the driver. The motor vehicle according to the invention can, in particular, be the motor vehicle mentioned in connection with the method according to the invention and / or in connection with the assistance system according to the invention, or correspond thereto. The motor vehicle according to the invention can therefore be configured for, in particular, automatically executing the method according to the invention.

[0041] The motor vehicle according to the invention and / or the assistance system according to the invention can be configured to automatically deactivate the assistance system or the described method, i.e., the corresponding function, in an efficiency or energy-saving mode. This can be applied, for example, in a battery-electric vehicle (BEV).

[0042] Further features of the invention can be derived from the following description of the figures and from the drawings. The features and combinations of features mentioned above in the description, as well as the features and combinations of features shown below in the description of the figures and / or in the figures alone, can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the invention.

[0043] The drawing shows: Fig. 1 is a schematic overview of a traffic situation to illustrate a method for warning a driver of a motor vehicle of hazards; and Fig. 2 a partial schematic representation of an interior of the motor vehicle to illustrate a possibility for warning the driver.

[0044] Identical or functionally equivalent elements are provided with the same reference numerals in the figures.

[0045] Fig. Figure 1 shows a partial schematic overview of a traffic scene or surrounding area with a section of a road that has a right lane 1, a center lane 2, and a left lane 3. These three lanes are intended for the same direction of travel. In the situation depicted here, a motor vehicle 4 and a rear-following vehicle 5 are traveling in the right lane 1.

[0046] The rear-following vehicle 5 can therefore be another vehicle that is following behind the motor vehicle 4. In the center lane 2, a center-following vehicle 6 is traveling, which, viewed in the direction of travel, is also traveling behind the motor vehicle 4. In the left lane 3, a left-following vehicle 7 is traveling in the same direction. For example, the left-following vehicle 7 is closer to the position of the motor vehicle 4, viewed in the direction of travel, but without having yet reached its position or overtaken it. Likewise, the left-following vehicle 7 could, for example, be at least substantially at the same height, i.e., at the same position in the direction of travel, as the motor vehicle 4.

[0047] The motor vehicle 4 is driven or controlled by a driver 8 in manual driving mode. For example, the driver 8 may currently be planning or intending a driving maneuver 9, indicated schematically here, to change from the right lane 1 to the center lane 2.

[0048] Current and future vehicles increasingly have displays, i.e. display devices, for showing the respective surroundings. For example, the motor vehicle 4 here has a plurality of mirror displays 10. On such displays or on the mirror displays 10, content, i.e. images and / or data, which were recorded or captured, for example, by means of an environmental sensor system 11 of the motor vehicle 4, can be shown. In the present case, however, it is not intended, or at least not intended, to display such content, i.e. camera images or video data recorded, for example, by means of a camera used as part of the environmental sensor system 11, unprocessed on the mirror displays 10. Rather, the attention of the driver 8 is intended to be specifically directed to specific content or parts of content, depending on the situation.corresponding objects in the surroundings of the motor vehicle 4 are steered which are important for it in the respective situation, for example, are relevant for the safe execution of the respective intended driving maneuver 9.

[0049] For this purpose, the motor vehicle 4 here also has a recording device 12 for recording the behavior of the driver 8, i.e. corresponding driver behavior data, and an assistance system 13. The assistance system 13 can be connected, for example, via an interface 14 to an on-board electrical system of the motor vehicle 4, indicated schematically here. Via this interface, the assistance system 13 can record data, such as, for example, environmental data provided by the environmental sensors 11 and driver behavior data provided by the recording device 12. To process this data, the assistance system 13 here has, for example, a processor 15 and a computer-readable data memory 16. The assistance system 13 can, for example, control the mirror displays 10 or an additional output device 17 of the motor vehicle 4 via the interface 14. The additional output device 17 can, for example, be a loudspeaker and / or a mechanical orhaptic actuator and / or a ventilation system or the like.

[0050] In the present case, the environmental sensors 11 or, respectively, by processing or evaluating corresponding environmental data by means of the assistance system 13 can observe the traffic to the rear or to the side from the perspective of the motor vehicle 4, i.e., objects located there, in particular the rear-following vehicle 5, the middle-following vehicle 6 and the left-following vehicle 7, can be detected. The environmental sensors 11 or the assistance system 13 can, for example, using artificial intelligence or an appropriately trained machine learning device, determine what type of objects are involved and how they are currently moving or behaving or will be behaving in the future. Furthermore, the assistance system 13 can, during the current journey or in the respective current situation and / or over a longer period of time, i.e., in particular over several journeys.recognize or determine a behavioral profile of the driver 8. The assistance system 13, in particular a correspondingly trained machine learning device provided therein, can combine the available data, i.e. in particular the driver behavior data and the respective current environmental data, and, based on the data in the respective situation, determine the driving maneuver 9 likely intended by the driver 8 and specifically relevant objects.

[0051] In the situation shown here, it can, for example, be automatically detected that the driver 8 probably wants to steer the motor vehicle 4 into the central lane 2 in accordance with the driving maneuver 9 indicated here. This probably intended driving maneuver 9 to change to the central lane 2 can, for example, be detected or predicted by the fact that the driver 8 has already steered the motor vehicle 4 to the left edge of the right lane 1 in the direction of travel and has, for example, already directed his gaze, in particular several times, to the mirror display 10 acting as the left side mirror or has, for example, already positioned his hand to more easily reach a lever or control element for activating the left direction indicator of the motor vehicle 4. For example, the assistance system 13 orwhose machine learning facility has learned from similar previous behaviors or behavior patterns of driver 8 that this combination of behaviors typically precedes a left lane change.

[0052] Furthermore, it can be detected, for example, that the left-following vehicle 7 has activated its turn signal in the direction of the central lane 2 and that the central following vehicle 6 has activated its brake lights. The latter can be detected, for example, by a correspondingly red-illuminated ground area in the rear area of ​​the central following vehicle 6 and / or by corresponding reflections or the like. From this, it can be concluded that the left-following vehicle 7 is likely to change into the central lane 2 and that the central following vehicle 6 will brake to enable this without a collision. The rear-following vehicle 5, on the other hand, is likely to continue driving in the right-hand lane 1, since it shows no signs of deviating behavior and there is insufficient space, for example, to change into the central lane 2.

[0053] Overall, the assistance system 13 can thus determine that, in the situation depicted here, the rear-following vehicle 5 and the center-following vehicle 6 are unlikely to interfere with the motor vehicle 4 during the execution of the driving maneuver 9, but that the left-following vehicle 7 represents or could pose a potential collision risk. Accordingly, the assistance system 13 can classify the rear-following vehicle 5 and the center-following vehicle 6 as secondary objects 18 that are not relevant for the safe execution of the driving maneuver 9. The left-following vehicle 7, on the other hand, can be classified as a hazardous object 19 that is relevant for the driving maneuver 9.

[0054] Since at least one relevant hazardous object 19 has been detected, the assistance system 13 can point this out or warn the driver 8.

[0055] To illustrate this, Fig.2 shows a partial schematic representation of an interior of the motor vehicle 4. Visible here, in particular, is a mirror display 10, which has a central interior rearview mirror area and, to the sides thereof, side mirror areas 20. Several objects in the vicinity of the motor vehicle 4 are visible therein, which, as described, may each have been classified as secondary objects 18 or as hazardous objects 19.

[0056] In this case, the detected relevant hazardous object 19 is marked by a warning sign 21 at the request of the assistance system 13. The warning sign 21 not only identifies the relevant hazardous object 19 as such, but also indicates its direction of movement or its approach to the motor vehicle 4, for example, by a corresponding arrow and, if applicable, a corresponding coloring or color coding of the warning sign 21.

[0057] Furthermore, the assistance system 13 can, for example, activate or control the additional output device 17 - simultaneously with the output of the warning 21 or with a delay - in order to output a further indication to the driver 8 regarding the hazardous object 19 or the detected hazardous situation. For example, a sequence of corresponding warnings 21 or the warning 21 and one or more further warnings can be initiated or output according to a detected danger level or upon a detected increase in the danger level in the respective current situation. A corresponding indication can be individually dependent on the respective situation and / or the driver 8 or their behavior and / or preferences or learned reactions and / or corresponding user settings.The reactions of the assistance system 13 can thus, for example, be based on experience learned from observing the driver 8 or be adapted or carried out as specified in a user profile of the driver 8. Thus, in different situations and / or for different drivers 8, for example, an optical or visual warning 21 can be issued first and, for example, if the driver 8 fails to react to it, an acoustic and / or haptic additional warning can be issued, or, for example, both the visual or optical warning 21 and at least one other additional warning can be issued initially.

[0058] If the warning or further information is an acoustic indication, its output and / or design can depend on the properties of the relevant object about which the driver is to be warned. Such properties can include, for example, the position and / or speed of the object relative to the motor vehicle and / or the direction in which the object is approaching the motor vehicle and / or the type of object. For example, an acoustic warning can be generated or output in such a way that a corresponding sound source appears to the driver to be at the position of the relevant object and / or to be moving in accordance with the actual movement of the relevant object. 3D or surround effects or functions or binaural 3D audio synthesis can be used for this purpose.Likewise, a sound sequence or tone order of the acoustic warning or further information can be adapted or adjusted accordingly depending on the characteristics of the respective relevant object.

[0059] Overall, the examples described show how digital side mirrors and the content marked therein can be used to increase safety in traffic. List of reference symbols 1 right lane 2 central lanes 3 left lane 4 Motor vehicle 5 rear-following vehicle 6 middle follower vehicle 7 Left-following vehicle 8 drivers 9 driving maneuvers 10 Mirror display 11 Environmental sensors 12 Recording device 13 Assistance system 14 Interface 15 processor 16 data storage 17 Additional output device 18 Secondary object 19 Dangerous object 20 side mirror areas 21 Warning

Claims

[1] Method for warning a driver (8) of a motor vehicle (4) of dangers in its surroundings, wherein during the travel of the motor vehicle (4) - the surroundings of the motor vehicle (4) are continuously monitored for objects (5, 6, 7) located there, - the driver’s behaviour (8) is continuously recorded, - based on the recorded behaviour of the driver (8), a driving manoeuvre (9) likely intended by the driver is determined, - those objects (5, 6, 7) detected in the surroundings of the motor vehicle (4) which are relevant for this driving manoeuvre (9) are determined, - a warning (21) is issued only on these specific relevant objects (7, 19), and a) driver behavior data indicating the behavior of the driver (8) of the motor vehicle (4) and corresponding driving maneuver data indicating driving maneuvers (9) actually performed by the driver (8) according to recorded driver behavior data are provided as training data for a machine learning device by an assistance system (13) for the motor vehicle (4) set up to carry out the method, and an output of the machine learning device trained therewith is used to determine or indicate the respectively expected intended driving maneuver (9), and / or b) the assistance system (13) for the motor vehicle (4) set up to carry out the method records routes travelled by the driver (8) over several journeys of the motor vehicle (4) as part of the driver behaviour data and a resulting movement profile of the driver (8) is taken into account to determine the respective likely intended driving manoeuvre (9). [2] Assistance system (13) for a motor vehicle (4), comprising an input interface (14) for detecting environmental data and driver behavior data indicating a behavior of a driver (8) of the motor vehicle (4), a data processing device (15, 16) for processing the detected data to determine a likely intended driving maneuver (9) and relevant objects (7, 19) in the environment and to generate a control signal to generate a warning (21) only for precisely these relevant objects (7, 19), and an output interface (14) for outputting the control signal, wherein the assistance system (13) is set up to carry out the method according to claim 1. [3] Assistance system (13) according to claim 2, characterized by that the assistance system (13) is designed to record, as part of the driver behavior data, data from which current movements and / or operating actions of the driver (8) during the current journey can be derived. [4] Assistance system (13) according to one of claims 2 to 3, characterized by that in variant b) the movement profile of the driver (8) is learned by means of machine learning. [5] Assistance system (13) according to one of claims 2 to 4, characterized by that the assistance system (13) is designed to use the environmental data to predict an expected future movement behavior of detected objects (5, 6, 7) in the environment of the motor vehicle (4) relative to the latter and to determine the relevant objects (7, 19) based on the respective prediction. [6] Assistance system (13) according to one of claims 2 to 5, characterized by in that the assistance system (13) is designed to control a mirror display (10) of the motor vehicle (4) to display a respective marking on the relevant objects (7, 19) visible in the mirror display in order to generate the warning (21). [7] Assistance system (13) according to one of claims 2 to 6, characterized by that the assistance system (13) is configured to automatically increase an intensity of the warning (21) compared to a respective initial display as soon as a predetermined criterion is met, in particular if no corresponding change in the behavior of the driver (8) or the likely intended driving maneuver (9) is detected in response to the warning (21). [8] Assistance system (13) according to one of claims 2 to 7, characterized bythat the assistance system (13) is designed to automatically control at least one further device for outputting a further warning which addresses a different sense of the driver (8) as soon as a predetermined criterion is met, in particular if no corresponding change in the behavior of the driver (8) or the likely intended driving maneuver (9) is detected in response to the warning (21). [9] Motor vehicle (4), comprising an environmental sensor system (11) for detecting objects (5, 6, 7) in the surroundings of the motor vehicle (4), a detection device (12) for detecting a behavior of the driver (8) of the motor vehicle (4), an assistance system (13) according to one of claims 2 to 8 and an output device (10, 17) coupled thereto for outputting warnings (21) to the driver (8).

Images