Motor vehicle and methods for representing surrounding objects and / or road users

Abstract

Method for displaying environmental objects and / or road users (10) to a user of a vehicle (1), wherein the vehicle has a communication device (2) which enables communication with infrastructure facilities and / or road users with regard to environmental objects and / or road users (10), in particular a Car2X unit, and at least one sensor device (4) which is suitable for detecting the environmental objects and / or road users (10), comprising the steps: - Determining the presence of the surrounding object and / or road user (10) using the communication device (2); - Output of at least one piece of initial information (I1), in particular visual information, to the user, which is characteristic of the surrounding object and / or the road user whose presence has been detected by the communication device (2); - Detection of the surrounding object and / or the road user (10) by means of at least one sensor device (4); - Output of at least one, in particular visually perceptible, second piece of information (I2) to the user, which is characteristic of the environmental object detected by the at least one sensor device (4) and / or the road user (10) detected by the at least one sensor device.

Description

[0001] The present invention relates to a method for displaying environmental objects and / or road users to a motor vehicle user and to a motor vehicle suitable for applying such a method. Numerous such methods are known in the prior art. For example, it is known that a warning message is issued to a motor vehicle user when, for instance, a vehicle overtaking on the right or left approaches.

[0002] It is also known that motor vehicles have various devices to detect, for example, objects in the environment or other road users. This is particularly relevant when such objects or road users cannot be readily perceived by a driver, such as a motorcyclist who is in a vehicle's blind spot or outside the driver's field of vision.

[0003] In addition, methods and devices are already known from the prior art which enable communication between a specific motor vehicle - hereinafter referred to as the ego vehicle - and other road users.

[0004] Furthermore, it is known in the prior art that objects can be highlighted for a user or driver using so-called augmented reality (AR) without requiring them to take their eyes off the road. AR and / or HUD (Head-Up Display) technology can also be used to direct the driver's gaze to draw attention to specific road users.

[0005] In this way, for example, in conjunction with a transmission technology such as Car2X technology, road users such as cyclists who are hidden by other objects can also be displayed before they cross the path of the ego vehicle.

[0006] Using Car2X technology, information is transmitted at a maximum frequency of 10 Hz according to current standards. When objects emitting CAMs (CAM stands for "Cooperative Awareness Messages") or VAMs (VAM stands for "VRU Awareness Message") are displayed, their position is also determined using GNSS (global navigation satellite system).

[0007] Both of these factors represent sources of error, especially when it comes to accurately determining the position of a road user such as a cyclist. Even with the application of suitable prediction algorithms, a visual highlight (for example, a frame) of a road user such as a cyclist can deviate so significantly from their actual position that the highlight would not appear at the road user's actual location.

[0008] In contrast, the representation of objects using local vehicle sensors (e.g., cameras, radar, lidar) has the advantage that the position relative to the vehicle is captured with high frequency and high spatial resolution. However, it is not possible to detect objects that are outside the vehicle's direct line of sight (LOS). A radar system is an exception in this respect, specifically for materials that are transparent to radar waves.

[0009] The present invention is based on the objective of achieving improved perceptibility of other (environmental) objects and / or road users. In addition, it aims to increase driving safety.

[0010] A driver assistance system and a driver assistance module are known from US document 2006 / 0055525 A1. This document mentions both the detection of road users using sensors and vehicle-to-vehicle communication.

[0011] From US patent 2023 / 0211776 A1, a method for determining attribute values ​​of obstacle-forming vehicles in vehicle communication is known.

[0012] The above-mentioned objectives are achieved according to the invention by the subject matter of the independent claims. Advantageous embodiments and further developments are the subject matter of the dependent claims.

[0013] Several detection scenarios can be distinguished. It is possible that certain objects and / or road users can only be detected by a Car2X system, but not with (precise) location. Other (environmental) objects can be detected by the vehicle's own sensors, usually with high accuracy.

[0014] Other road users can also be perceived by the vehicle occupant, for example through the windshield or side windows. In the following, the objects perceived by the occupant are essentially those that the occupant can perceive without having to take their eyes off the road in front of them, particularly by looking through the windshield.

[0015] In the following, the term "environmental objects" refers to objects that are not part of the ego-vehicle but belong to the environment of the ego-vehicle and towards which the ego-vehicle moves.

[0016] In an inventive method for displaying environmental objects and road users to a user of a motor vehicle, wherein the motor vehicle has a communication device which enables communication with infrastructure facilities and / or road users with regard to environmental objects and / or road users, in particular a Car2X unit and at least one first sensor device which is suitable for detecting the environmental objects and / or road users, the following steps are carried out: - Determining the presence of the surrounding object and / or road user by means of the communication device; - Output (in particular by means of an output device) of at least one piece of initial information, in particular visual (or visually perceptible), to the user, which is characteristic of the surrounding object and / or the road user whose presence has been detected by the communication device. - Detection of the surrounding object and / or the road user by means of at least one sensor device. - Preferably, a comparison is made between the environmental object and / or road user detected (and in particular recognized) by the at least one sensor device and the environmental object and / or road user for which the first piece of information is characteristic and which is / was output to the user. In particular, a fusion and / or attribution of information between the object (environmental object and / or road user) detected by the Car2X unit and the object (environmental object and / or road user) detected and / or recognized by the sensor device (of the vehicle) is performed. This advantageously avoids a duplicate representation of the object (environmental object and / or road user). - Output of at least one second piece of information to the user, which is particularly visually perceptible and is characteristic of the environmental object detected by the sensor device and / or the road user detected by the sensor device.

[0017] Preferably, the first piece of information indicates the presence of an object in the environment and / or a road user in the vicinity of the ego-vehicle. Furthermore, the information can also be characteristic of the type of object in the environment and / or the road user. An example of such information would be the display of a message like "Caution cyclist" or "Caution cyclist approaching from the right".

[0018] Alternatively, the initial piece of information could also be a warning, such as a stylized warning triangle. This warning triangle could depict the type of object in the surroundings and / or the road user, such as a (stylized) cyclist.

[0019] The initial information could also already contain a (rough) direction of movement of the surrounding object and / or road user (especially relative to the ego-vehicle).

[0020] Preferably, the first piece of information can be a preliminary warning. Ideally, this initial information is provided without a more precise location or position. This initial information can specifically indicate the presence of an object in the vicinity of the ego-vehicle and / or another road user.

[0021] It would also be possible for the first piece of information to be output with a rough geometric assignment, for example with the information that an environmental object or a road user is approaching from the right or from the left.

[0022] The second piece of information, containing a location, is preferably displayed. However, it is also possible for the second piece of information to highlight an object in the user's environment and / or a road user that is already visible. For example, if a cyclist is already within the user's field of vision, this cyclist can be made more visible by highlighting them, such as by framing them. In this case, the second piece of information highlights an object in the environment and / or a road user that is already visible.

[0023] If the surrounding object and / or the road user is already detected by a sensor device but is not yet visible to the road user, the second piece of information can also contain a warning and display the surrounding object and / or the road user to the user, in particular with a (now more precise) position indication, for example if the surrounding object and / or the road user is in a blind spot for the user or would only be visible through the side windows.

[0024] In another preferred method, it is checked whether the surrounding object and / or the road user whose presence was detected by the communication direction matches the surrounding object and / or the road user detected by the sensor device.

[0025] Car2X data can be V2V communication data (vehicle-to-vehicle communication data), V2I communication data (vehicle-to-infrastructure communication data), V2N communication data (vehicle-to-network communication data) and / or V2P (vehicle-to-pedestrian).

[0026] Vehicle-to-vehicle communication data may include, for example, communication data transmitted in particular according to the ETSI ITS G5 communication technology.

[0027] Such vehicle-to-vehicle communication data may include, for example, sensor data (or derived quantities) that were recorded and / or collected and / or generated by an (on-board sensor) environmental sensing device of the road user sending the vehicle-to-vehicle communication data.

[0028] For example, so-called "Collective Perception Messages" (CPM) can be used as such vehicle-to-vehicle communication data. In particular, such CPM messages can contain information about the kinematic dynamics and position, as well as other attributes of objects (e.g., vehicles, pedestrians, animals, and others) that are detected by environmental perception devices (especially those of the road user sending the vehicle-to-vehicle communication data), such as radar, lidar, and camera(s), to which the transmitting road user has access. The Collective Perception Message (CPM) enables, in particular, the interoperable exchange of basic information about the transmitting road user (necessary for interpreting the transmitted data), their sensory capabilities, the perceived objects, and / or road-related perception areas.

[0029] CPMs are generated quasi-periodically, as determined by the CPM generation event. This offers the advantage that the transmitted data volume is kept small by triggering the transmission of CPMs based on the perception of a new object and / or a new attribute.

[0030] Such vehicle-to-vehicle communication data offers the advantage that information (especially regarding movement behavior or position and / or time) relating to road users can be recorded and thus taken into account in the local swarm data (and therefore global swarm movement data), which themselves do not have the property of sending (or receiving) vehicle-to-vehicle communication data.

[0031] The vehicle-to-vehicle communication data is preferably in the form of so-called "Cooperative Awareness Messages" (CAMs). This vehicle-to-vehicle communication data is used to cyclically or periodically inform nearby communication partners (such as other road users) of its presence (by sending the CAM message).

[0032] The use of CAM messages offers the advantage that a new CAM message is sent if the distance to the sending position of the previous CAM message exceeds a predefined distance (e.g., 4 m). This allows for closely timed transmission of presence information and reliable warning of the vehicle user.

[0033] Furthermore, the use of CAM messages offers the advantage that if a road user changes course, i.e., if their orientation in the World Geodetic System of 1984 (WGS84) changes by more than 4° compared to the last transmitted CAM, a new CAM is sent. This allows for very precise tracking of a road user's movement path, particularly during turning maneuvers and cornering, and thus provides reliable warnings to the vehicle user regarding the other road user.

[0034] Furthermore, the use of CAM messages offers the advantage that if the current speed of the road user exceeds the speed value sent in the last CAM by more than 0.5 m / s, a new CAM is sent, so that even large acceleration and / or deceleration values ​​are recorded and can be transmitted to the motor vehicle.

[0035] Preferably, the transmitted vehicle-to-vehicle communication data includes information relating to the time of generation of the vehicle-to-vehicle communication data and the position of the road user.

[0036] Preferably, the transmitted vehicle-to-vehicle communication data additionally or alternatively includes information relating to at least one geometric and / or motion parameter of the road user, which is selected from a group of parameters including a direction of travel, a direction of travel, an orientation of the road user, a speed, a length, a width of the road user, a longitudinal acceleration, a curvature, a yaw rate, an acceleration control, a lane position, a lateral acceleration, a steering wheel angle, a vertical acceleration, a power class, a toll zone and the like, as well as combinations thereof.

[0037] Preferably, the transmitted vehicle-to-vehicle communication data additionally or alternatively includes information regarding the role of the road user (e.g., special vehicle, ordinary passenger vehicle, etc.). This offers the advantage that information about the size and type of road user enables a reliable determination of the potential hazard and thus a reliable warning of the vehicle user.

[0038] Preferably, the transmitted vehicle-to-vehicle communication data also includes, or alternatively includes, information relating to a path history. This offers the advantage that, for example, the position of the road user can be determined even more precisely, thus enabling even more reliable warnings for the vehicle user.

[0039] For example, the Car2X data can be used to determine the nature of the surrounding object and / or road user (cyclist, pedestrian, traffic sign, construction site, etc.). The sensor system can also detect and determine the type of road user or surrounding object. These respective findings are then preferably compared.

[0040] For example, if this comparison shows that both the communication device and the sensor device have detected a cyclist, and in particular the same cyclist, this may lead to the conclusion that it is the same environmental object or the same road user.

[0041] This result can be achieved, for example, by a processor unit comparing a captured image with reference images. A vehicle license plate can also be recognized. This check is preferably performed when an object in the environment and / or a road user is initially only detectable via the communication device and then moves into the driver's field of vision and / or the detection range of a sensor unit.

[0042] Here too, the sensor setup is particularly preferred, or data from the sensor direction is used to identify the object.

[0043] In another preferred method, the output of the first piece of information is followed by the output of the second piece of information, or both the first and second pieces of information are output. In particular, this occurs after it has been determined that the surrounding object and / or the road user whose presence was detected by the communication device corresponds to the object detected by the sensor device, in particular the surrounding object and / or the road user detected by the sensor device.

[0044] Therefore, identification of the surrounding object detected by the Car2X device or the road user identified by the Car2X device is particularly preferred.

[0045] Preferably, identification can be based on different categories, such as the type of road user, for example, whether it is a cyclist, motorcyclist, pedestrian, or the like. The surrounding object can also be differentiated by type, for example, a traffic light, a bollard, or the like.

[0046] In another preferred method, the first and second pieces of information are displayed to the user via a display device, and in particular via the same display device. The second piece of information could, for example, be a highlighting of an object in the surroundings or a road user already within the user's field of vision.

[0047] For example, a road user who is already visible can be displayed with an outline. In another preferred method, the first piece of information and / or the second piece of information is displayed to the user using augmented reality. For instance, if a cyclist is detected by the Car2X system, this can be displayed to the user using augmented reality, especially since the cyclist is not yet visible. Arrows and a stylized image of a cyclist can be displayed to the user via the display device.

[0048] Furthermore, for example, a visible vehicle can also be displayed with a border, i.e., with a border created by augmented reality.

[0049] In another preferred method, the first and second pieces of information are displayed to the user via a head-up display (HUD). It is also possible that only parts of the second piece of information are displayed, while other parts can be directly visible to the user.

[0050] In another preferred method, the first piece of information and / or the second piece of information is also displayed to the user, at least temporarily, by means of a second display device. This second display device could be, for example, a display in the dashboard, a touchscreen, or a display that is visible through a steering wheel.

[0051] In another preferred method, the position of the surrounding object and / or the road user is detected, in particular by the sensor device. Furthermore, it is also possible that several sensor devices are provided, which, however, function differently and / or are arranged at different positions on the ego-vehicle and thus enable a more precise detection of surrounding objects and / or road users.

[0052] Preferably, an (exact) position relative to the ego-vehicle can be recorded. For example, the distance between an object in the surroundings or another road user and the ego-vehicle can be determined. In particular, the front end of the vehicle can be used as a reference point.

[0053] Another preferred method also takes data from a navigation system into account. For example, navigation data might indicate that a specific object in the environment or a road user detected via Car2X is still obscured by buildings, trees, or similar obstacles. This information can also be displayed to the user.

[0054] In another preferred method, the process transitions from the output and / or display of the first information to the output and / or display of the second information by means of an animation.

[0055] For example, the first piece of information can be hidden while the second piece of information is displayed.

[0056] Preferably, a switch from the output of the first information to the output of the second information can be achieved by "fading out over black" or by fading out the output or display of the first information and fading in the second information.

[0057] Additionally or alternatively, a switch or transition from the output of the first piece of information to the output of the second piece of information can be achieved through morphing. In this way, the representation of the first piece of information can be deformed or transformed (especially continuously) into the representation of the second piece of information. For example, a cyclist icon (representation of the first piece of information) can be deformed into a bounding rectangle and / or a (bounding) bar and / or a (bounding) circle and the like, and / or into a (at least partially and preferably completely) line-curve-shaped element.

[0058] Preferably, an animation is used in which the output of the first piece of information transitions into the output of the second piece of information, depending on the criticality, particularly with regard to the object (surrounding object and / or road user) detected by the Car2X unit and identified and / or recorded by at least one sensor device of the vehicle. This allows for a variation of the animation depending on the criticality. For example, a more noticeable animation can be selected and displayed for a more critical event (such as an imminent accident).

[0059] Preferably, a variety of different animation types are specified, and depending on a recorded and / or determined criticality, one of the animation types is selected and used to output the animation of a transition from the first output information to the second.

[0060] The criticality refers in particular to the object (environmental object and / or road user) detected (by the Car2X unit) and identified and / or recorded by at least one sensor device of the vehicle, preferably to (traffic) safety in relation to this object.

[0061] It is conceivable that the individual animation types among the multitude of animation types differ from one another in a geometric extent of the animation output and / or in its color design and / or in a temporal progression of the output (longer or shorter animation duration) and / or a (changing) color gradient, a sequence of color changes which lead to a blinking effect, in a temporal variation (blinking effect), in a change in the orientation of the output element and / or in a shape of the output element and / or (at least sectionally) in an output path along which an element is output during the animation, in a type of movement during the output (linear movements, movement along a circle, and the like).

[0062] The criticality can be determined by a criticality assessment device in the vehicle, particularly a processor-based one, especially depending on the environmental data acquired by at least one sensor device in the vehicle. However, it is also conceivable that a warning signal regarding criticality is received via a communication device in the vehicle, such as the Car2X unit.

[0063] It is conceivable that different criticality levels are predefined. Preferably, a criticality level is determined (e.g., by the criticality assessment device), particularly with regard to the (identified and / or determined and / or recorded) object (environmental object and / or road user) with respect to which the first piece of information is output and with respect to which the second piece of information is to be output. Preferably, for the display of the animation, an animation type is selected from the multitude of different animation types depending on the criticality level. Each criticality level can be assigned a specific (predefined) animation type.

[0064] Preferably, when the second piece of information is output, a horizontal shift occurs compared to the (last previous) output of the first piece of information. This horizontal shift of the object during animation can be useful if the position of the surrounding object and / or road user detected (via the Car2X unit) does not correspond to the position of the surrounding object and / or road user detected and / or recognized by the vehicle's sensor system. The horizontal shift offers the advantage of compensating for the fact that the object detected by the Car2X unit (surrounding object and / or road user) may be offset from the object detected and / or recognized by the vehicle's sensor system.The object (environmental object and / or road user) detected and / or recognized by the sensor device of the (motor) vehicle therefore appears either earlier or later at the edge of the obstruction of vision than expected.

[0065] Preferably, the initial information regarding the detected object (surrounding object and / or road user), particularly at a visibility edge and / or visibility contour, is "held" until it becomes visible or perceptible / detectable to the sensor system of the (motor) vehicle (in the case that the Car2X object appears too early). This also advantageously accounts for the fact that the Car2X object is / may be offset relative to the sensor object. It therefore appears either earlier or later than expected at the edge of the visual obstruction.

[0066] Preferably, the visibility edge and / or the visibility contour is predefined (and stored in a permanent memory device of the vehicle) and / or predefinable. It is also conceivable that the visibility edge and / or the visibility contour is adaptable to the respective sensor device used.

[0067] Preferably, the output position at which the first information is output can change (as the output time progresses, especially continuously) and can be adapted to a (continuously) changing relative position of the (motor) vehicle and the detected object (environmental object and / or road user).

[0068] Preferably, the output position of at least one output area, and preferably of the entire output area of ​​the first piece of information, does not exceed the specified and / or specifiable visibility edge and / or visibility contour.

[0069] Preferably, the output position at which the second piece of information is output can change (as the output time progresses, especially continuously) and can be adapted to a (continuously) changing relative position of the (motor) vehicle and the detected object (environmental object and / or road user).

[0070] Preferably, the output position of at least one output area, and preferably of the entire output area of ​​the second piece of information, does not exceed the specified and / or specifiable visibility edge and / or visibility contour.

[0071] Preferably, the first piece of information is output on a different side of the visibility edge and / or visibility contour relative to the second piece of information. In other words, the output position of the first piece of information lies (exclusively) on the first side of the visibility edge and / or visibility contour of a (fundamentally possible) output area of ​​the vehicle's output device. Additionally or alternatively, the output position of the second piece of information lies (exclusively) on a different side of the visibility edge and / or visibility contour of a (fundamentally possible) output area of ​​the vehicle's output device.

[0072] Preferably, the visibility edge and / or the visibility contour forms a boundary of a (maximum possible) perception area or detection area of ​​the at least one sensor device and / or a plurality of sensor devices of the (motor) vehicle.

[0073] It is conceivable that a color representation in which the second piece of information is displayed could be adapted to the color representation of the first piece of information, at least when the second piece of information is shown or displayed for the first time. For example, the same color representation could be chosen. Furthermore, the criticality of the information being presented to the user is also taken into account when choosing the color.

[0074] In other preferred methods, the second piece of information includes a visual element that highlights an environmental object or road user that is actually visible to the user directly, i.e., without any display devices. This could be a border around the actually detected or visible object, a color highlight, or something similar. It is conceivable that the object itself is not displayed.

[0075] Preferably, when transitioning from the first piece of information to the second piece of information (instead of the first), the display location (within the screen) and preferably also the display method are changed. In particular, when transitioning from the first piece of information to the second piece of information (instead of the first), both the display location and the display method are changed.

[0076] Preferably, the first piece of information and / or the second piece of information is output depending on a (human) viewing angle and / or field of vision (e.g., that of the driver). In particular, it is preferably determined, depending on the driver's viewing angle and / or field of vision, at which position the second piece of information should be output, so that the driver's view of the object is not obstructed (and the object is merely highlighted by the output of the second piece of information).

[0077] Only the second piece of information is given preference, not the first piece of information, depending on a (human) perspective (such as that of the driver).

[0078] Preferably, an assessment is made as to whether the environmental object detected (by means of the Car2X unit) and / or the road user detected (by means of the Car2X unit) and / or the environmental object and / or road user detected by the vehicle's at least one sensor device is (already) within the driver's or user's field of vision. Preferably, the second piece of information is only output once the environmental object and / or road user detected by the vehicle's at least one sensor device is within the driver's or user's field of vision.

[0079] This offers the advantage that an adjustment can be made to the driver's field of vision if this deviates from the sensor system or the at least one sensor device (and preferably the sensor device(s) by which the surrounding object and / or the road user was detected and / or recognized) due to a different position and / or opening angle.

[0080] Preferably, the first piece of information is output until the surrounding object (already detected and / or recognized by at least one sensor device of the vehicle) and / or the road user (already detected and / or recognized by at least one sensor device of the vehicle) is within the driver's / user's field of vision. In other words, the second piece of information is only output (instead of the first) when the assessment shows that the object (surrounding object and / or road user) detected (by the Car2X unit) and captured and / or recognized by at least one sensor device is within the driver's / user's field of vision.

[0081] Preferably, a change (preferably conversion) of the output of the first information to the output of the second information is triggered by the (especially within the framework of the assessment described above) recognized entry of the object (environmental object and / or road user) detected (by means of the Car2X unit) and captured and / or recognized by the at least one sensor device into the field of vision of the driver or user of the output device.

[0082] Preferably, a viewing angle and / or a field of view of the driver or user can be detected and / or determined by a field-of-view detection device (especially of the vehicle), such as an interior camera of the vehicle.

[0083] It is also conceivable that field-of-view data, which indicate a typical field of view and / or viewing angle for a driver or user, are stored in a memory device (of the vehicle) and are in particular predefined or (e.g. by the driver or user) predefined, are taken into account to assess whether the environmental object detected (by means of the Car2X unit) and / or the road user detected (by means of the Car2X unit) and / or the environmental object and / or road user detected and / or recognized by means of at least one sensor device of the vehicle is (already) in a field of view of the driver or user.

[0084] It is also conceivable that the field of view data is determined (automatically) by a field of view data determination device, in particular a processor-based one, depending on a (set) seat height and / or set positioning of the (external and / or rear) mirrors.

[0085] Preferably, the second piece of information is output to a different location than the first. Preferably, the second piece of information is output with greater spatial accuracy relative to the detected object than the first piece of information relative to the detected object. This takes into account that vehicle-mounted sensor systems (typically) have or can offer a much higher resolution and a greater update rate than is achieved by Car2X messages.

[0086] Preferably, the second piece of information is output in such a way that its output position correlates with a (current) determined and / or recorded (absolute and / or relative to the vehicle) position of the detected and / or recorded and / or determined object (surrounding object and / or road user). "More precise" is preferably understood to mean that there is a stronger correlation. For example, the output location of the second piece of information could be adjusted more precisely and / or more frequently to the current or an updated (relative) position of the object than the output of the first piece of information. It would be conceivable, for instance, that the output location of the second piece of information is continuously and / or at regular intervals adjusted to the current or an updated (relative) position of the object, while the output location of the first piece of information is not adjusted (or is adjusted less frequently than the first piece of information).

[0087] It is conceivable that the first piece of information is issued at a (essentially) fixed location.

[0088] Preferably, the position of the environmental object and / or road user detected by the Car2X unit (i.e., the Car2X object) is determined, and a characteristic value for this position is output to the driver. It is conceivable that the output position of the first piece of information correlates with the determined position of the detected environmental object and / or road user. However, it is also conceivable that, for example, an arrow displayed as part of the output of the first piece of information indicates to the driver a direction in which the detected environmental object and / or road user is to be expected.Additionally or alternatively, it is conceivable that, depending on the determined position of the detected object and / or road user, at least one output area is selected from a multitude of predefined output areas (e.g., a left edge area of ​​an output device and a right edge area of ​​an output device) to output the initial information. This allows the driver to infer the position or direction of the detected object from the selected output area. In particular, this output occurs during a period in which the object and / or road user detected by the Car2X unit has not yet been detected by at least one (or any) of the vehicle's sensor devices.

[0089] It is also conceivable that the position of the object detected by the Car2X unit could be output along with map data (e.g., for navigation).

[0090] Preferably, the output position or location of the second piece of information is repeatedly and / or continuously adjusted (to the current relative position of the vehicle with respect to the object). In this way, the second piece of information is preferably carried along with the object. Alternatively, the output position or location of the first piece of information is preferably not adjusted; instead, and particularly preferably, the first piece of information is output at the same location for the entire duration of the output or display.

[0091] Preferably, the system checks whether an environmental object and / or road user relevant to the user (especially with regard to driving safety) is located in an area of ​​the vehicle's surroundings that is not perceptible (visible) or detectable by the user and / or at least one (preferably all) of the vehicle's sensor devices. Therefore, initial information regarding an object (environmental object and / or road user) detected in this non-perceptible (visible) or non-detectable area is preferably only issued if the detected object is relevant to the user (especially with regard to driving safety). A relevant object can be understood as an object against which the user must be warned to increase road safety and / or driving safety.This offers the advantage that a (static) warning can be avoided when no relevant objects are nearby, so as not to strain the acceptance (and attention demanded by issuing the warning) of drivers.

[0092] The sensor device is particularly preferred if it is selected from a group of sensor devices which includes cameras, video cameras, radar, lidar, infrared sensors, image cameras, ultrasonic sensors and the like.

[0093] In addition, the data can also be recorded and / or captured using multiple sensor devices. For example, radar can be used for certain environmental objects and image capture devices for others.

[0094] In another preferred method, the data from each sensor device is selected to display the output. This can depend, for example, on the quality of the data or on other factors such as the time of day. Thus, data from different sensor devices can be used at night than during the day, or the data can be selected depending on the vehicle speed.

[0095] In another preferred method, at least one of the pieces of information takes into account a driving parameter of the ego-vehicle. In particular, this can include speed, acceleration, direction of travel, or a change of direction, such as a lane change or a turn.

[0096] Furthermore, an additional warning is preferably issued indicating that there is a risk of collision if a certain driving style is maintained.

[0097] In addition, it is also possible for a vehicle control system to intervene in the vehicle or the driving process, such as braking or swerving.

[0098] For example, if a sensor system determines, taking into account the driving parameters, that a collision may occur shortly, the vehicle control system can initiate (full) braking.

[0099] In order to take advantage of the benefits of the two technologies described above, i.e. the communication device and the sensor device, and to overcome their disadvantages, according to the invention a change and in particular a dynamic change of the display of relevant road users such as the aforementioned cyclists can be carried out.

[0100] The basis is preferably an environment model that processes both Car2X data and sensor data from the vehicle or the vehicle's sensor devices and preferably makes this data available to the driver assistance systems. This is mentioned, for example, in DE 10 2020 207 065 B3. In this respect, and in particular with regard to the method described therein for combining object information about one or more objects in a vehicle's environment from different data sources and / or determining a (cooperative) environment model, reference is made to patent specification DE 10 2020 207 065 B3, the contents of which are hereby incorporated into this application.

[0101] The combination of vehicle-local sensors and Car2X preferably enables matching / attributing of (environmental) object data from both sources and advantageously the determination of the exact time when an object already received via the communication device and in particular via Car2X enters the sensor cone of the at least one sensor device and in particular of camera / Lidar / Radar / Ultrasonic sensors.

[0102] As soon as an object existing in the environment model is to be displayed to the driver due to the driving situation, the display method in the AR HUD is preferably defined depending on the availability of the data in the environment model: Objects that exist solely via Car2X and are therefore not yet within the field of vision of the vehicle's sensors (and / or the driver) are preferably represented as stylized objects, whereby an exact representation of the hidden position is neither necessary nor desirable due to the aforementioned disadvantages.

[0103] As soon as an environmental object enters the detection range of a sensor device (e.g., camera) and can be attributed / fused with the Car2X data, the object display or the display of the road user is preferably switched so that the real environmental object and / or the road user is highlighted by the AR HUD.

[0104] As a result, the surrounding object can now be precisely tracked by the vehicle's sensors and displayed to the user. To make the transition more appealing to the user, and / or to, for example, mask an offset during the switchover, it can be designed with an animation.

[0105] Therefore, it is preferable to attribute and / or fuse data from the communication device on the one hand and data from the sensor device(s) on the other hand in the environment model mentioned above.

[0106] Preferably, switching occurs (possibly with animation) between a particularly stylized and non-localized representation for pure Car2X environment objects and / or road users to a localized highlighting of objects that have been detected by local sensors or by at least one of the sensor devices (of the vehicle) and / or which are visually perceptible to the user of the vehicle.

[0107] The present invention further relates to a motor vehicle, wherein the motor vehicle has a communication device which enables communication with infrastructure facilities and / or road users with regard to surrounding objects and / or road users, in particular a Car2X unit.

[0108] Furthermore, the motor vehicle has at least one first sensor device which is suitable and intended for detecting surrounding objects and / or road users.

[0109] Furthermore, the motor vehicle has an output device which is suitable and intended for outputting at least one initial piece of information, in particular visual information, to the user, which is characteristic of the surrounding object and / or a road user whose presence has been detected by the communication device.

[0110] Furthermore, the motor vehicle has an output device which is suitable and intended for outputting at least one second piece of information, in particular visually perceptible, to the user, wherein this information is characteristic of the object detected by the sensor device and / or the road user detected by the sensor device.

[0111] Preferably, the output device, which is suitable and intended for outputting at least one, in particular visual, first piece of information to the user, which is characteristic of the surrounding object and / or a road user whose presence has been detected by the communication device, and the output device, which is suitable and intended for outputting at least one, in particular visual, perceptible second piece of information to the user, wherein this information is characteristic of the object detected by the sensor device and / or the road user detected by the sensor device, is the same output device, in particular a HUD.

[0112] For example, it is possible that a road user is registered with a location identifier, and this is emphasized in the display device, for instance, by means of a border. This border then represents the second piece of information.

[0113] The second piece of information is particularly preferred if it is characteristic of the environmental object detected by the sensor device and / or the road user detected by the sensor device. In particular, this second piece of information is characteristic of a position of this environmental object and / or road user.

[0114] According to the invention, the motor vehicle further comprises a switching device which is suitable and intended to effect (especially under given conditions) a transition from an output of the first information to the output of the second information.

[0115] Preferably, the second piece of information is an emphasis function or a warning notice, which highlights an object actually visible to the user, for example through a windshield, and which advantageously informs about a possible imminent accident or collision or generally about a dangerous situation.

[0116] Preferably, the output device is a visual display device, and in particular a head-up display. In a further preferred embodiment, the motor vehicle has a verification device which is suitable and intended to determine whether an environmental object and / or a road user whose presence has been detected by means of the communication direction corresponds to an environmental object and / or a road user that has been detected by the sensor device.

[0117] Preferably, only in this case of a match is the transition made from the output of the first piece of information to the output of the second piece of information.

[0118] Preferably, the inspection device can then determine what kind of object the surrounding object whose presence has been detected is.

[0119] In another preferred embodiment, an identification device is provided which is suitable for identifying the object whose presence has been detected.

[0120] Furthermore, it is also possible that the sensor device is suitable and designed to record video sequences of a road user's movement. Based on these video sequences, it would be possible, for example, to reconstruct the sequence of events in the event of an accident.

[0121] Particularly preferred is a comparison device which compares an environmental object or a detected road user with sensor data from at least one sensor device.

[0122] For example, the communication device detects the presence of a motorcycle and checks whether a camera has also recorded a motorcycle or whether the user themselves sees a motorcycle. In this case, it is possible to perform the aforementioned switching of the displays.

[0123] In a further preferred embodiment, it is checked or verifiable whether a camera has also recorded the motorcycle (or the detected surrounding object and / or the detected road user). Only in this case does a transition in the display preferably occur.

[0124] In a special configuration, a warning message can also be issued if there is a deviation in the detected and / or recorded surrounding objects and / or road users.

[0125] Preferably, the (motor) vehicle is equipped, suitable, and / or intended to carry out the above-described method and all process steps already described above in connection with the method, either individually or in combination. Conversely, the method can be equipped with all the features described in connection with the vehicle, either individually or in combination, or use elements described in this context.

[0126] The vehicle in question may in particular be a (motorized) road vehicle.

[0127] A vehicle can be a motor vehicle, which is in particular a driver-operated vehicle ("driver only"), a semi-autonomous vehicle, an autonomous vehicle (for example, at autonomy levels 3, 4, or 5 (according to the SAE J3016 standard)), or a self-driving vehicle. Autonomy level 5 refers to fully automated vehicles. The vehicle can also be a driverless transport system. In this case, the vehicle can be driven by a driver or drive autonomously. Furthermore, in addition to a road vehicle, the vehicle can also be an air taxi, an aircraft, or another means of transport or vehicle type, such as an aircraft, watercraft, or rail vehicle.

[0128] The present invention further relates to a computer program or computer program product, comprising program means, in particular a program code, which represents or encodes at least some of the and preferably all of the process steps of the method according to the invention and preferably one of the described preferred embodiments and is designed for execution by a processor device.

[0129] The present invention further relates to a data storage device on which at least one embodiment of the computer program according to the invention or a preferred embodiment of the computer program is stored.

[0130] Further advantages and embodiments can be seen from the attached drawings:

[0131] It shows: Fig. 1. A schematic representation of the display of initial information; and Fig. 2. A schematic representation of the display of secondary information

[0132] Fig. Figure 1 shows a schematic representation of a motor vehicle 1 according to the invention. This vehicle has an output device 6 in the form of a head-up display 6. The reference numeral 2 roughly schematically identifies a communication device and, in particular, a Car2X device, which is suitable and intended for communicating with other road users and / or surrounding objects.

[0133] Reference numeral 4 designates a sensor device suitable and intended for detecting surrounding objects and / or other road users. Preferably, the motor vehicle has several such sensor devices, which may also be of different types.

[0134] Reference symbol I1 indicates initial information. This is output taking into account data from communication device 2. In this case, this information indicates that a road user has been detected in the vicinity of the motor vehicle.

[0135] Reference symbol S4 denotes the representation of a stylized cyclist. The three arrows P1, P2, P3 indicate the direction of movement of this cyclist. The number of these arrows can be chosen, for example, depending on the relative speed of the road user in relation to the motor vehicle.

[0136] Reference sign 12 identifies a motor vehicle's steering wheel. Reference sign 22 identifies a second display device, which can preferably also output information characteristic of the surrounding object and / or the road user. For example, information I3 can indicate the type of road user, and information I4 can display a warning message.

[0137] Reference symbol 26 indicates an additional operable display device, such as a touchscreen. The user can preferably use this device to configure the information display described here, for example, to set the type of warning messages.

[0138] Fig. 2 shows that in Fig.Figure 1 shows a motor vehicle in a second operating situation. Here, a road user, such as a cyclist, was detected by a sensor device 4. The cyclist may also be visible to the user through the windshield.

[0139] The reference sign I2 indicates a second piece of information, which in this case is a highlighting of road user 10 in the form of a frame, which is also output via the output device 6.

[0140] A warning message 15 can be issued via the further output device 22.

[0141] The applicant reserves the right to claim all features disclosed in the application documents as essential to the invention, provided they are novel individually or in combination compared to the prior art. It is further noted that the individual figures also describe features which may be advantageous on their own. A person skilled in the art will immediately recognize that a particular feature described in a figure may be advantageous even without incorporating other features from that figure. Furthermore, a person skilled in the art will recognize that advantages may also arise from a combination of several features shown in individual or different figures. Reference symbol list 1 motor vehicle 2 Communication device 4 Sensor device 6 Output device, HUD 10 road users 13 stylized representations 14 Warning 15 Warning 16 Windscreen 22 additional output facilities 24 steering wheel I1 first information I2 second information I3, I4 further information P1, P2, P3 Display symbols of the first information S4 animated rendering QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2006 / 0055525 A1

[0010] US 2023 / 0211776 A1

[0011] DE 10 2020 207 065 B3

[0100]

Claims

[1] Method for displaying environmental objects and / or road users (10) to a user of a vehicle (1), wherein the vehicle has a communication device (2) which enables communication with infrastructure facilities and / or road users with regard to environmental objects and / or road users (10), in particular a Car2X unit, and at least one sensor device (4) which is suitable for detecting the environmental objects and / or road users (10), comprising the steps: - Determining the presence of the surrounding object and / or road user (10) using the communication device (2); - Output of at least one piece of initial information (I1), in particular visual information, to the user, which is characteristic of the surrounding object and / or the road user whose presence has been detected by the communication device (2); - Detection of the surrounding object and / or the road user (10) by means of at least one sensor device (4); - Output of at least one, in particular visually perceptible, second piece of information (I2) to the user, which is characteristic of the environmental object detected by the at least one sensor device (4) and / or the road user (10) detected by the at least one sensor device. [2] Method according to claim 1, characterized by , that it is checked whether the object and / or road user (10) whose presence was detected by the communication device (2) matches the object and / or road user (10) detected by the sensor device (4). [3] Method according to at least one of the preceding claims, characterized by, that the output of the first information (I1) is skipped to the output of the second information (I2) or both the first information (I1) and the second information (I2) are output after it has been determined that the object and / or road user (10) whose presence has been detected by the communication device (2) matches the object and / or road user (10) detected by the sensor device (4). [4] Method according to at least one of the preceding claims, characterized by , that the first information (I1) and the second information (I2) are output to the user via a display device (6) and in particular via the same display device. [5] Method according to at least one of the preceding claims, characterized by, that when transitioning from the output of the first piece of information to the output of the second piece of information, a change in the output location occurs. [6] Method according to at least one of the preceding claims, characterized by , that the presentation of the first piece of information (I1) and / or the presentation of the second piece of information (I2) is output to the user using augmented reality and / or by means of a head-up display (HUD). [7] Method according to at least one of the preceding claims, characterized by , that at least one position of the object and / or road user is detected in particular by the sensor device (4) and at least one piece of information is output to the user which is characteristic of the position of the object and / or road user (10). [8] Method according to at least one of the preceding claims, characterized by, that the transition is made from the output and / or display of the first piece of information (I1) to the output and / or display of the second piece of information (I2) by means of an animation. [9] Method according to at least one of the preceding claims, characterized by , that at least one of the information (I1, I2) driving parameters of the motor vehicle is taken into account when the output is generated. [10] Motor vehicle (1) wherein the motor vehicle has a communication device (2) which enables communication with infrastructure facilities and / or road users with regard to environment objects and / or road users (10), in particular a Car2X unit, and has at least one first sensor device (4) which is suitable and intended for detecting environment objects and / or road users (10), and wherein the motor vehicle further has an output device (6) which is suitable and intended for outputting at least one, in particular visual, first piece of information (I1) to the user which is characteristic of an environment object and / or a road user whose presence has been detected by the communication device (2), and the motor vehicle (1) has an output device (6),which is suitable and intended for outputting at least one second piece of information (I2), in particular visually perceptible, to the user, which is characteristic of the object detected by the sensor device (4) and / or the road user (10) detected by the sensor device, , characterized by that the motor vehicle further comprises a switching device (12) which is suitable and intended to effect a transition from an output of the first information (I1) to the output of the second information (I2) under specified conditions.

Images