Method for assisting a user of an assistance system, assistance system and vehicle equipped with such a system

Abstract

To provide a supporting system, and a method of supporting a user of such a supporting system, including a display (1) displaying a representation of one or more objects (3, 4, 5) within an environment of the display (1).SOLUTION: In a first step, information about a display environment is determined (S1) and a presence of an object in the environment is determined (S2). Next, an environment representation including a representation of an object (3) determined in the environment is generated. As to at least one of the determined and displayed objects (3'), a starting point (7) of an indicator line (8) is calculated from a direction of the object with respect to the display (1; S5); and for each determined starting point (7), the indicator line (8) is drawn connecting the starting point (7) and the displayed representation (3') of a real-world object (S6).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for assisting a user of an assistance system in perceiving the environment, and to the respective assistance system, in particular an advanced driver assistance system for a vehicle. [Background technology]

[0002] To improve the safety of traffic participants, various assistance systems have been developed over the past few years. These systems typically use sensors to physically sense the system's environment, e.g., the vehicle's environment. To assist the user, such systems generate a representation of the environment. Often, such systems are designed to generate behavior suggestions based on the representation. Other approaches aim only to improve the user's perception of the environment. These systems display a representation of the environment by displaying objects sensed in the environment on a display, which is mounted on a vehicle held by or operated by the user. Often, the sensed objects or related information are displayed using an augmented reality display. Such approaches are known, for example, from U.S. Pat. No. 9,690,104 or EP 3570225 A1.

[0003] One problem with known systems is that unless the system is designed for fully automated driving, the user still needs to observe the environment themselves. Even with a level 2 automated system, the driver still needs to observe the vehicle's surroundings and conditions in case of abnormal situations. Thus, adding additional information to what the user would need to perceive in the environment anyway leads to an increase in information volume. Perceiving the environmental details needed to make the right decisions to safely engage in a dynamic environment, such as congested traffic, is a difficult task. Thus, simply adding information that needs to be additionally perceived by the user may even distract the user, and thus may be detrimental to the desired effect of improving safety. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,690,104 [Patent Document 2] EP 3570225 A1 Summary of the Invention [Problem to be solved by the invention]

[0005] It should be noted that this problem is not limited to assistance systems that require the driver to necessarily align the real world with a representation on a display. Event navigation systems may use sensed external information to enhance the impression of realism, which automatically leads the driver, or users of such systems in general, to try to identify real-world objects with corresponding representations on a display. Thus, the object of the present invention is to reduce the user's perception of the environment by showing correspondences between objects in the assisted user's environment (real-world objects) that are included as icons in the representations used in the assistance system. [Means for solving the problem]

[0006] This object is achieved by a method, an assistance system and a corresponding vehicle comprising such an assistance system according to the invention.

[0007] According to the present invention, a method for assisting a user of an assistance system including a display displaying one or more icons corresponding to objects (or images thereof) in the display's environment comprises first obtaining information about the display's environment, which is the vehicle's environment if the display is mounted on a vehicle; determining the presence of objects in the environment using a sensor; and displaying a representation of the environment including the determined objects in the environment on the display; then determining, for at least one of the determined and displayed objects, a starting point of an indicator line from the orientation of the object relative to the display; finally, displaying, for each determined starting point, an indicator line connecting the starting point to the displayed object. The assistance system of the present invention comprises a display, a respective sensor, and a processing unit configured to perform the method steps described above.

[0008] The general concept of the present invention is that a user of an assistance system will easily recognize information about the environment displayed on a display when he or she can easily identify objects in the display's environment with the respective icons (images) that represent those objects on the display. As a result, the time required by the user to align the information directly perceived from observing the environment with the information provided by the display is reduced. This results in an improved overall situation understanding by the user and ultimately in fewer errors made by the user. If such an assistance system is dedicated to assisting traffic participants, for example, when operating a vehicle equipped with such an assistance system, this leads to improved safety.

[0009] The dependent claims define advantageous embodiments of the method and assistance system.

[0010] According to a preferred embodiment, the starting point is calculated as the intersection point between the outer edge of the display and a surface area that includes a direction vector extending from a vertical axis passing through a reference point of the display and pointing from the reference point toward the real-world object for which the indicator line is to be displayed. Identifying the starting point and displaying the indicator line as described above allows the user to intuitively identify real-world objects in the vehicle's environment with corresponding object representations (icons, images) on the display. Locating the starting point at the edge of the display in a very intuitive way ties the real-world object to its representation on the display, thus easily allowing the user to identify the real-world object with the corresponding icon or simplified object representation on the display.

[0011] The identification of objects in the real-world environment of the assistance system with the represented object on the display is further improved when a direction vector is calculated from a reference point in the display to point to the center of the determined object.

[0012] A further improvement is achieved if, for at least one determined and displayed object, an indication bar corresponding to the perceptible horizontal extent of the determined object is displayed along the outer edge of the display, the indication bar including the starting point. This is particularly useful when the indication line itself may be ambiguous due to the presence of multiple objects in the real world. The indication bar allows for direct recognition of the horizontal dimension of the object in the real world, which gives the user of the assistance system a further indication of the correspondence between the real-world object and its representation on the display.

[0013] Preferably, the indication bar extends from a first intersection point of the edge of the display and a first bounding surface area to a second intersection point of the edge of the display and a second bounding surface area, the first bounding surface area including a first direction vector extending from a vertical axis passing through the reference point and pointing toward a first outermost perceptible boundary of the determined object in a horizontal direction from the reference point, and the second bounding surface area including a second direction vector extending from a vertical axis passing through the reference point and pointing toward an opposite second outermost perceptible boundary of the determined object in a horizontal direction from the reference point. Because the ends of the indication bar are defined by the first and second intersection points determined in a manner similar to the start point, the indication bar is intuitively recognized by a user, like a projection of a real-world object onto the display.

[0014] According to an advantageous embodiment, if the boundary surface areas of two determined objects match, immediately adjacent indicator bars are displayed using distinguishable characteristics. Thus, even if one of the objects in the display or assistance system environment is partially obscured by another object and both objects are displayed in the representation, the arrangement of the indicator bars with distinguishable characteristics makes it easy to determine which indicator bar belongs to which real-world object. It should be noted that the horizontal extent of the obscured object is assumed to be limited by the horizontal extent of the obscuring object. Consequently, in the case of partially obscured objects, the rightmost surface area of ​​one object will necessarily match the leftmost surface area of ​​the other object, or vice versa. The characteristic may be, inter alia, the color, brightness, or pattern of the indicator bar.

[0015] An assistance system according to the present invention comprises a display controlled by a processing unit, the processing unit being configured to: obtain information about the environment of the display sensed by a sensor; determine the presence of objects in the environment; cause the display to display a representation of the environment including the determined objects in the environment; determine, for at least one of the determined and displayed objects, a starting point of an indication line from the orientation of the object relative to the display; and cause the display to display, for each determined starting point, an indication line connecting the starting point to the representation of the object.

[0016] Additionally, if an indication bar is to be displayed, the processing unit is further configured to cause the display to display, for at least one determined and displayed object, an indication bar along an outer edge of the display corresponding to the perceivable horizontal extent of the determined object, the indication bar including a starting point.

[0017] Further aspects and details will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 illustrates the resulting display, which allows easy recognition of correspondences between real-world objects and their representations on the display. [Figure 2] FIG. 10 is a schematic diagram for explaining a method for determining the starting point of an instruction line. [Figure 3] FIG. 3 is an enlarged view of FIG. [Figure 4] FIG. 10 is a diagram illustrating a method for determining an indication bar corresponding to an object in the real world. [Figure 5] 1 is a simplified block diagram including the main components of an assistance system according to the present invention; [Figure 6] 2 is a simplified flow chart detailing method steps according to the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0019] FIG. 1 shows a display 1, on which a representation of the environment of the display 1 is displayed. The environment of the display 1 is sensed by one or more sensors. Based on this, a processing unit calculates the representation to be displayed from the sensor signals. Typically, such a display 1 is installed in a vehicle, and a driver of the vehicle is assisted in perceiving the environment and thus in taking correct decisions to drive the vehicle safely. In the following description, it is assumed that the display 1 is installed in such a vehicle and is part of an advanced driver assistance system.

[0020] Vehicles equipped with driver assistance systems including such displays are well known in the art, so to simplify the drawings, only the vehicle's display 1 is shown.

[0021] In the situation shown in Figure 1, which is used to explain the functioning of the method and assistance system of the present invention, several vehicles can be detected in the environment of the display 1. From Figure 1 it can be seen that the vehicle in which the display 1 is installed is driving in a lane of a road, which is separated from adjacent lanes by lane markers 2 on the left and right.

[0022] In the adjacent lane to the right, a vehicle 3 can be identified, and further, a truck 5 following the vehicle 3 can be identified. In addition, another vehicle 4 is traveling ahead in the own lane. Of course, the object sensed by the sensor and determined from the sensor signals by post-processing all the sensor signals is not limited to a vehicle. Many systems are available on the market that are capable of identifying multiple different objects from the sensor signals. The sensor may be, for example, a radar sensor, a LIDAR sensor, an ultrasonic sensor, etc. Therefore, the identification of objects in the signals received from the respective sensors is known in the art, and a description thereof will be omitted for the sake of brevity.

[0023] The description given below will refer to vehicle 3 in order to avoid limiting the description to vehicles traveling in the same lane as the ego-vehicle. However, the description given is equally valid for any object that can be determined in the environment of display 1. It should be noted that the following description will all refer to vehicles, since in particular advantageous aspects become immediately apparent here. Perceiving moving objects in the environment of a traffic participant is more difficult than identifying static elements. Nevertheless, the invention is applicable to all objects that can be determined in the environment of display 1.

[0024] Sensors, not shown in Figure 1, physically sense the vehicle's environment, and therefore the environment of display 1. From the sensor outputs, a processor calculates the determined object's position in a coordinate system, which has a fixed relationship to display 1. The fixed relationship of display 1 and the sensors to each other is known from the system design. Thus, whatever coordinate system the position of an object in the environment is calculated in, the position can be transformed into the coordinate system of display 1. This will be explained in more detail with reference to Figures 2 and 3.

[0025] From the sensor outputs, a representation of the environment on display 1 is calculated. Objects that can be determined from the sensor outputs are displayed on display 1 as corresponding icons. Depending on the resolution of display 1 and the processing capabilities of the overall system, it is also possible to use images representing real-world objects instead of icons. In the example shown in Figure 1, vehicles 3, 4 and truck 5 are shown as grey surfaces 3', 4' and 5'. Also shown is a lane marker 2' indicating the lane in which the ego-vehicle (and also vehicle 4) is traveling. The position of the ego-vehicle is indicated by a further icon 6 containing an arrow to indicate the direction of travel of the ego-vehicle.

[0026] According to the invention, a user of the assistance system is assisted in identifying correspondences between perceptible objects in the real world outside the display 1 and their corresponding icons displayed on the display 1. In the present example, this is explained with reference to a vehicle 3 as an example.

[0027] Based on the sensor output, a starting point 7 is calculated by a processor (not shown in FIG. 1 ). The starting point 7 is calculated to be located on an edge 10 of the display 1. The starting point 7 is then connected to an icon 3′ corresponding to a real-world object, i.e., a vehicle 3, by drawing an indicator line 8. The indicator line 8 between the starting point 7 and the icon 3′ is preferably an arrow whose end terminates within the grey area of ​​the icon 3′.

[0028] As will be explained further below, the position of the starting point 7 on the edge 10 of the display 1 is determined so that the indication line 8 resembles a direct connection from the real-world vehicle 3 to the corresponding icon 3'. As briefly discussed above, the position of the real-world vehicle 3 in the coordinate system of the display device 1, which is the relative position of the vehicle 3 with respect to the display 1, is known.

[0029] In addition to the indicator line 8, in a preferred embodiment an indicator bar 9 is displayed on the display 1. The indicator bar 9 extends along the outer edge 10 of the display 1. The length of the extension along the edge 10 of the display 1 corresponds to the horizontal dimension of the corresponding real-world object, as this dimension is perceptible by the user of the assistance system. This means that for objects that are at a greater distance to the display 1, only a short indicator bar 9 is displayed on the display 1. If the same object is closer to the display 1, it is represented by an indicator bar 9 with a longer extension.

[0030] Depending on the relative position of the real-world object with respect to the display 1, the indicator bar 9 may extend across multiple edges 10 of the display 1, which may have, for example, a rectangular shape. In the illustrated embodiment, this can be seen by the indicator bar in the upper right corner of the display 1, representing the track 5. However, as will become clear from the following description, all calculations determining the start point 7 and end point of the indicator bar 9 refer to the same reference point on the display 1. This reference point is the center of the icon 6, the point where all axes of the coordinate system intersect. The absolute orientation of the coordinate system of the display 1 is irrelevant to the calculation of the start point 7 and the indicator bar 9, as long as the following two conditions are met: a) One axis must extend vertically while the other two span a horizontal plane. b) The orientation of the coordinate system is static.

[0031] For a first description of the method of the invention, reference is now made to Figure 2. As mentioned above, the description is limited to a single environmental object sensed by a sensor, namely a vehicle 3. Of course, the representation of the displayed environment in Figure 2 is adapted so that only the icon 3' and the lane markers 2' are shown.

[0032] As can be seen in Figure 2, it is not absolutely necessary for the indicator line 8 to extend to the middle of the surface of the icon 3', it is sufficient that it points unambiguously towards the icon 3' so that the user can identify the icon 3' with the respective real-world vehicle 3.

[0033] To enable the display 1 to output a screen such as that shown in Figure 2, it is necessary to determine the location of a starting point 7 on an edge 10 of the display 1. This technique will be explained with reference to the top of Figure 2 and the enlarged portion of Figure 3.

[0034] The vehicle 3 is equipped with an assistance system of the present invention, which includes at least one sensor that allows determining the direction in which an object in the environment of the vehicle is located. The position determination can be performed in a first coordinate system, which in this case is indicated by an arrow pointing in the direction of travel of the vehicle 3. With respect to this coordinate system, the angle α is determined. In general, the position and orientation of the display 1 and the coordinate system used to determine the position of the vehicle 3 have a fixed relationship, so that the position of the vehicle 3 in the coordinate system of the display 1 can be easily determined. For the purposes of understanding the present invention, it is sufficient to assume that the coordinate systems of the sensors and the display 1 coincide. It is noted that this assumption is made only for ease of understanding and does not limit generality.

[0035] As described above, the coordinate system of the display 1 is arranged so that all three axes of the coordinate system pass through the reference point corresponding to the position of the host vehicle on the display 1, which is the origin of the coordinate system.

[0036] Once the orientation of the vehicle 3 relative to the display 1 is known, a surface area S0 is determined that includes a direction vector d extending from a vertical axis y passing through a reference point on the display 1 and pointing towards the vehicle 3.

[0037] This surface area S0 intersects with the display 1 and thus has an intersection point with one edge 10 of the display 1. To avoid the surface area S0 having a second intersection point, the surface area S0 extends only in the direction of a direction vector d that points from the vertical axis y towards the real-world object, the vehicle 3. The only requirement that must be met is that the display 1 and the vertical axis y intersect at a single point, which means that the vertical axis does not lie in the plane of the display 1.

[0038] Since the position of the icon 3′ is known in advance and the position of the starting point 7 on the edge 10 of the display 1 has now been determined, the indicator line 8 can be drawn. Several different approaches are possible for selecting the second end point (the end of the arrow) of the indicator line 8. First, the indicator line 8 can connect the starting point 7 to the center of the area of ​​the icon 3′. Second, the indicator line 8 can extend along the intersection line between the surface area S0 and the display 1. This intersection line necessarily extends from the starting point 7 toward the origin of the coordinate system of the display 1. Assuming that the user of the assistance system intuitively identifies his or her own position with the icon 6 representing the vehicle and thus with the position of the reference point on the display 1, this provides the most natural approach for easily identifying real-world objects and the environment of the display 1 with their corresponding icons. However, the length of the indicator line 8 may be selected based on design considerations, for example to avoid obscuring other objects. Thus, the indicator line 8 may extend to the center of the icon or may simply point toward the boundary of the icon area.

[0039] It should be noted that well-known image processing techniques may be applied to determine the direction vector d. For example, the contour of an image of a real-world object in the environment representation can be generated from the data received from the sensor, and the coordinates at the center of the contour area can be selected as the ends of the direction vector. Other alternatives may also be considered.

[0040] An advantageous embodiment is shown in Figure 4. In addition to the indicator line 8, an indicator bar 9 is displayed for each determined and displayed object, e.g., a vehicle 3 displayed in the representation of the environment on the display 1. For illustrative purposes, the indicator line 8 as well as the surface area S0 used to calculate the starting point 7 have been omitted from the drawing.

[0041] To determine the first and second end points of the indicator bar 9, the first surface area S L , the second surface area S RThe intersections 11, 12 of the first surface area S with the edge 10 of the display 1 are determined. L and a second surface area S R The calculation of the intersection points 11, 12 of the surface area S is similar to the calculation of the starting point 7 described with reference to Figures 2 and 3. L , S R is the vertical axis y of the coordinate system of display 1 and the direction vector d L , d R The direction vector d L , d R points from the origin of the coordinate system of the display 1 towards the leftmost and rightmost points of the outline of the real-world object. The resulting indication bar 9 thus corresponds to the extent of the real-world object in the horizontal direction.

[0042] 5 shows a simplified block diagram with a display 1, one or more sensors 13, and a processor 14 as the main components of an assistance system 15 of the present invention. The assistance system 15 is typically mounted on a vehicle, and sensors 13 available to other assistance systems of such a vehicle may be shared by the various assistance systems.

[0043] The sensor outputs are fed to a processor 14 which, based on the information contained in the sensor signals, calculates a representation of the environment in a representation generation unit 16. According to the invention, also based on the sensor outputs, a surface calculation unit 17 calculates one or more surface areas SO, S as explained above. L , S R and supplies information about these surfaces to the start point / end point calculation unit 18.

[0044] It should be noted that units 16, 17 and 18 may be realized as software modules, the software being processed by the same processor 14. The "processor" 14 may also consist of several individual processors combined into one processing unit. Furthermore, the coordinates of the display 1 in its coordinate system are known and stored in the support system 15. Thus, the surface areas S0, S1, S2, S3, S4, S5, S6, S7, S8, S9, S10, S11, S12, S13, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S26, S27, S28, S29, S30, S31, S32, S33, S44, S45, S56, S57, S58, S59, S60, S61, S62, S63, L , S R Based on these surface areas S0, S L , S R It is easy to calculate the intersection (intersection point as well as the intersection line) between the surface of display 1. It should be noted that whenever the description given regarding the present invention refers to "display 1", it means the display surface visible to the user, and not the entire display device.

[0045] As already explained above, coordinate transformation may be performed in a pre-processing step after the environment of the display 1 has been sensed by the sensor 13. Such coordinate transformation is only necessary if the coordinate system used for the sensor 13 (and thus for determining the relative positions of objects in the real world) is not the same as the coordinate system of the display 1. If the position of the sensor 13 and the display 1 are very close, the same coordinate system may be used and no transformation of coordinates is necessary.

[0046] FIG. 6 shows a simplified flow chart illustrating the main method steps according to the present invention as detailed above.

[0047] First, in step S1, the environment of the display 1 is sensed using sensors. Then, in step S2, objects in the environment are determined from the sensor output. In step S3, direction vectors d, d pointing towards the center of the object or towards horizontal extrema that identify, for example, the leftmost and rightmost boundaries of a traffic object are determined. L , d R is calculated.

[0048] In step S4, the determined direction vectors d, d L , d R The surface areas S0, S L , S R These surface areas are then used in step S5 to calculate intersection points located on the outer edge 10 of the display 1. Finally, in step S6, an indicator line 8 is displayed for each calculated start point 7. Each pair of first and second end points is assigned a surface area S L , S R If the calculation is made from the corresponding indicator bar 9,

[0049] According to the present invention, it is very easy for a user of the display 1 to identify real-world objects with corresponding icons or other representations of the real-world objects displayed on the display 1. Thus, the time required by the user to review all relevant information to successfully and accurately estimate, for example, a traffic situation in a dynamic environment is significantly reduced, thereby avoiding the undesirable effects of providing additional information.

[0050] A preferred application area is integration into advanced driver assistance systems, or more generally, assistance systems used in vehicles. Such systems may use on-board sensors and displays. However, standalone solutions are also conceivable. For example, handheld devices used for navigation may be equipped with respective sensors so that such standalone devices can also utilize the present invention. The latter may represent a significant safety improvement for pedestrians who tend to look at their mobile devices while navigating to an unknown destination. According to the present invention, an object perceived out of the corner of their eye can be easily identified with a corresponding icon on the display. Even without having to look up to obtain complete information about this particular object in the real world, pedestrians can at least make a basic estimate of the relevance of each real-world object.

[0051] Specifically, the combination of indicator lines and indicator bars allows for the identification of real-world objects at a glance. As detailed above, calculations of surface area, all passing through the origin of the coordinate system of display 1, and therefore through the reference point of display 1, result in the use of indicator bars to scale the dimensions of real-world objects at the edges of display 1, which are intuitively perceived as the boundary between the "virtual world" and the "real world."

Claims

1. 1. A method of assisting a user of an assistance system (15) including a display (1) displaying a representation of one or more objects (3, 4, 5) in the environment of said display (1), comprising: a method step (S1) of obtaining information about the environment of said display; a method step (S2) of determining the presence of an object (3, 4, 5) in said environment; - displaying a representation of said environment including said objects (3, 4, 5) determined within said environment; a method step (S5) of determining, for said determined and displayed at least one object (3), a starting point (7) of an indication line (8) from the orientation of said at least one object (3) relative to said display (1), said starting point (7) being located on an outer edge (10) of said display (1); a method step (S6) of displaying, for each determined starting point (7), said indication line (8) connecting said starting point (7) and the displayed representation (3') of said at least one object (3).

2. The starting point (7) is a surface area (S) that extends from the outer edge (10) of the display (1) and a vertical axis y passing through a reference point of the display (1) and includes a direction vector d pointing from the reference point towards the at least one object (3) on which the indication line (8) is to be displayed. ０ ), which is calculated as the intersection between The method of claim 1.

3. the direction vector d is calculated as pointing from the reference point in the display (1) to the center of the determined at least one object (3); The method of claim 2.

4. For the at least one object (3), an indication bar (9) corresponding to the determined scaled horizontal extent of the at least one object (3) is displayed along the outer edge (10) of the display (1), the indication bar (9) including the starting point (7). The method of claim 1.

5. The indicator bar (9) is positioned between the outer edge (10) of the display (1) and a first boundary surface area (S Ｌ ) from the intersection (11) of the outer edge (10) of the display (1) and a second boundary surface area (S Ｒ ) and the first boundary surface area (S L) extends from a vertical axis y passing through a reference point and is defined by a direction vector d pointing from the reference point towards the first outermost perceptible boundary of the determined at least one object (3) in the horizontal direction. Ｌ and said second boundary surface area (S Ｒ ) includes a direction vector extending from the vertical axis passing through the reference point and pointing from the reference point towards an opposite second outermost perceptible boundary of the determined at least one object (3) in the horizontal direction, The method of claim 4.

6. The boundary surface area S of the two determined objects Ｌ , SR coincide, the immediately adjacent indicator bar (9) is displayed using distinguishable characteristics, The method of claim 5.

7. 1. An assistance system comprising a display (1) controlled by a processing unit (14), the processing unit (14) being configured to: obtain information about an environment of the display (1); determine the presence of objects (3, 4, 5) in the environment; cause the display (1) to display a representation of the environment including the determined objects (3, 4, 5) in the environment; determine, for the at least one object (3), a starting point (7) of an indication line (8) from an orientation of the object (3) relative to the display (1), the starting point (7) being located on an outer edge (10) of the display (1); and cause the display (1) to display, for each determined starting point (7), the indication line (8) connecting the starting point (7) to a displayed representation (3') of the at least one object (3).

8. 8. The assistance system of claim 7, wherein the processing unit is further configured to cause the display (1) to display, for the at least one object (3), an indication bar (9) along an outer edge (10) of the display (1) corresponding to the determined perceptible horizontal extent of the at least one object (3), the indication bar (9) including the starting point (7).

9. A vehicle equipped with an assistance system (15) according to claim 7 or 8.

Images