METHOD FOR PREDICTING AT LEAST ONE FUTURE SPEED VECTOR AND / OR ONE FUTURE POSE OF A PEDESTRIAN

Description

[0001] The present invention relates to a method for predicting at least one future velocity vector and / or a future pose of a pedestrian within a prediction area. Furthermore, the present invention relates to a computer program that executes each step of the method, and to a machine-readable storage medium that stores the computer program. Finally, the invention relates to an electronic control unit configured to execute the method. State of the art

[0002] To prevent collisions between self-driving vehicles or robots and pedestrians, methods are used that predict future speed vectors and poses of pedestrians.

[0003] US Patent 9,233,659 B2 describes a collision warning system that uses a vehicle-mounted camera and a processor. A warning of a collision between the vehicle and a pedestrian is issued based on changes in the size of an image of the pedestrian captured by the camera. Temporal changes in the image and road markings captured in the image are taken into account to further validate the warning. This allows the system to detect lateral movement of the pedestrian when they cross a road marking or a curb.

[0004] WO 2015 / 177648 A1 describes a method for detecting pedestrians that can identify them in images. This method evaluates whether there is a risk of the pedestrian colliding with a vehicle. For this purpose, a future position of the pedestrian and the vehicle is extrapolated from their respective direction of movement and speed.

[0005] US Patent 2007 / 0230792 A1 describes a method for detecting whether a pedestrian might enter a vehicle's path. This involves identifying pedestrians within images recorded by a camera. They are then classified by comparing their location and trajectory with the curb line. If the classification indicates that the pedestrian is in the roadway, a collision warning is issued.

[0006] US Patent 2013 / 329958 A1 discloses a method for tracking individuals within the field of view of cameras. If a person moves out of the field of view of one camera, their direction and speed are used to predict in which other camera's field of view they are likely to reappear, allowing for a targeted search within that specific area. This involves searching for a particular individual within a group of people who has a distinctive appearance and tracking this individual instead of the actual target. The system records, once, the direction and distance of the actual target relative to the distinctive individual, and this relative position is also maintained in a blind spot between two cameras.

[0007] The article by D. Becker et al., "Vehicle and Pedestrian Collision Prevention System based on Smart Video Surveillance and C2I Communication," 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE, 2014, pages 3088-3093, discloses how to use the prediction of a pedestrian's future velocity vector to control a self-driving vehicle in such a way as to avoid a collision with the pedestrian. For this purpose, an ad-hoc network consisting of numerous stationary cameras positioned in the vicinity of the self-driving vehicle is created to detect a pedestrian and determine their direction of movement and speed.

[0008] CN 106 663 193 A describes a detection system for a motor vehicle. This system includes an image capture device that can detect a curb. Disclosure of the invention

[0009] The method for predicting at least one future velocity vector and / or pose of a pedestrian within a prediction area requires that the prediction process consider not only data about the pedestrian in question, but also a static map of the prediction area and the current velocity vectors of other pedestrians within that area. To effectively prevent a collision with a pedestrian, it is necessary not only to know their current position and velocity vector, which can be determined by sensors, but also to predict how the pedestrian's velocity vector and pose will change in the future in order to issue a timely collision warning.Particularly when the procedure is used to avoid a collision between a high-speed self-driving vehicle and a pedestrian, it is insufficient to know the pedestrian's current location, direction of travel, and speed. Future changes in the pedestrian's velocity vector can transform a situation previously considered safe into a potential collision scenario. In such cases, sufficient time must be available for the vehicle to decelerate or execute an evasive maneuver before a collision with the pedestrian occurs.

[0010] The map of the predicted area through which a vehicle will travel in the future is created by the vehicle's sensors. This map contains all the geographical features of the predicted area. A movement plan is then generated from this map, showing which sections of the predicted area, free from obstacles, are accessible to pedestrians.

[0011] While prior art methods only consider future changes in the velocity vector due to stationary obstacles in a stationary map or social information without information about the environment, the present method incorporates both a map and the movement of other pedestrians into the prediction. In particular, these pedestrians are not merely viewed as mobile obstacles, but interactions between them can also be included in the prediction.

[0012] Preferably, pedestrians within the prediction area are grouped together for forecasting purposes. From the current poses and velocity vectors of all members of a group, a current pose and velocity vector for the group as a whole can be determined. Pedestrians are grouped together, in particular, if their current velocity vectors indicate that they are moving as a group. This is especially true if the velocity vectors of all group members differ from each other by a maximum of a predefined threshold. It simplifies the execution of the method on an electronic computer or control unit if all group members can be treated uniformly in the prediction by using a common pose and velocity vector.

[0013] Furthermore, it is preferable to consider social interactions between pedestrians in the prediction. For pedestrians who are members of a group, such social interactions can consist particularly of attractive interactions that cause group members to never stray further than a predetermined distance from a central point of the group, the nearest group member, or the group leader. This can lead group members, when they have to avoid an obstacle, to choose not the shortest possible route, but instead a longer one that maintains group cohesion. Social interactions can also be relevant for prediction between pedestrians who are not part of a group. For example, a pedestrian might briefly stop to let another pedestrian pass or move aside to avoid them.A social interaction during an evasive maneuver can also consist, in particular, of both pedestrians performing an evasive maneuver, rather than one continuing on their way unhindered while the other avoids the other.

[0014] It remains preferred that each pedestrian be assigned a destination. This assignment can be made, in particular, based on their current velocity vector. The prediction takes into account that the pedestrian is moving towards this destination. Even if, for example, they are temporarily diverted from the direct path to this destination by social interactions within a group, by interactions with pedestrians who are not part of their group, or by other factors such as stationary obstacles, the prediction still assumes that they will, in the long run, always return to a path leading to this destination.

[0015] Preferably, the prediction is performed in several temporal substeps. In each of these substeps, all values ​​considered in the prediction of the velocity vectors are recalculated. Such values ​​include, for example, the velocity vectors and poses of other pedestrians and the social interactions with each of these pedestrians. In this way, a prediction is obtained in each temporal substep, with the predictions of the first substeps being very accurate and the accuracy decreasing as the prediction extends further into the future.

[0016] In each of the sub-steps, a future pose of the pedestrian can be determined. From all the determined future poses, it is particularly possible to create a movement map of the pedestrian, which can be used to plan the movement of a self-driving vehicle or a robot.

[0017] The current poses and velocity vectors of pedestrians can be determined, in particular, using at least one sensor selected from the group consisting of monocular sensors, stereo sensors, and depth sensors. A LiDAR sensor or an RGB-D camera, for example, is suitable as a depth sensor.

[0018] The computer program is configured to execute each step of the procedure, particularly when running on a computing device or an electronic control unit. Specifically, it includes a prediction module with a movement plan, tracklets, and social context information used for prediction. It allows for the implementation of different embodiments of the procedure in an electronic control unit without requiring any structural modifications. For this purpose, it is stored on a machine-readable storage medium.

[0019] By uploading the computer program to a conventional electronic control unit, the electronic control unit is obtained which is set up to predict a future velocity vector and / or a future pose of a pedestrian using the method. Brief description of the drawings

[0020] Exemplary embodiments of the invention are shown in the drawings and are explained in more detail in the following description. Fig. 1 schematically shows a system architecture of a self-driving vehicle which, by means of an embodiment of the method according to the invention, makes a prediction of at least one future velocity vector and a future pose of a pedestrian. Fig. 2 schematically shows the process of predicting a future velocity vector and a future pose of a pedestrian in an embodiment of the method according to the invention. Fig. 3 schematically depicts social interactions between several pedestrians, which are used in an embodiment of the method according to the invention for predicting at least one future velocity vector and a future pose of a pedestrian. Fig. 4 schematically shows the movement of a group of pedestrians relative to a self-driving vehicle in a method according to an embodiment of the invention.Figure 5 shows in a diagram the negative log probability as a function of the forecast duration in simulations according to comparative examples and an embodiment of the invention. Figure 6 shows in a diagram the modified Hausdorff distance as a function of the forecast duration in simulations according to comparative examples and an embodiment of the invention. Exemplary embodiments of the invention

[0021] In one embodiment of the method for predicting at least one future velocity vector and a future pose of a pedestrian in order to avoid a collision between the pedestrian and a self-driving vehicle, sensors on the vehicle create a map 11 of an area through which the vehicle will travel in the future. The map 11 contains all the geographical features of the area. The aim is to predict whether pedestrians within this area could enter the vehicle's path of travel, thus creating a risk of collision. Therefore, this area is referred to below as the prediction area. Images of the prediction area are created using the sensors, and people within the images are detected 12.In a prediction module 20, a movement plan 21 is generated from the map 11. This plan shows which sections of the prediction area, unblocked by obstacles, pedestrians can move through. The data collected about individual pedestrians are divided into tracklets 22 and social context information 23. The tracklets 22 comprise the current velocity vectors and poses of the pedestrians. Each velocity vector contains information about a pedestrian's direction of movement and their speed. The pose indicates their orientation. The social context information 23 is obtained through image analysis and allows for an assessment of which individual pedestrians are part of a group with a common destination. The movement plan 21, the tracklets 22, and the social context information 23 are then made available to the prediction module 30.The result of the prediction is transmitted to a control unit 40 of the self-driving vehicle to avoid a collision with pedestrians. It is also used to facilitate the next pedestrian detection process 12 in the recorded images.

[0022] In Fig. 2 The process of prediction 30 is described in detail. In step 31, the movement plan 21, the tracklets 22, and the social context information 23 are recorded, with individual pedestrians being grouped together based on the social context information 23. In the next step 32, the pedestrians' destinations are extrapolated based on their current velocity vectors. Several possible movement routes and destinations are considered for this purpose. In the next step 33, these are corrected using the movement plan 21 so that only 11 plausible destinations remain based on the map. The collected data on the current velocity vectors, current poses, and destinations of each pedestrian are then passed to a prediction function 50 in the next step 34. This function has four sub-steps.In a first sub-step 51, the path each pedestrian is likely to take within a given time interval is determined based on their destination, current velocity vector, and current pose. In a second sub-step 52, this path is corrected by considering each pedestrian's social interactions with neighboring pedestrians. These social interactions represent probable collision avoidance paths with other pedestrians based on typical pedestrian movement patterns. In a third sub-step 53, a further correction is made based on social interactions between group members. These social interactions influence movement in such a way that groups are not separated.In a fourth sub-step 54, based on the results of the first three sub-steps 51, 52, and 53, the current poses and velocity vectors are replaced by predicted future poses and velocity vectors at the end of the time interval. The future poses thus determined are entered into a motion map 35 and linked to the time interval on which the prediction is based. Sub-steps 51 to 54 are then repeated, but this time the future velocity vectors and poses of the pedestrians, determined in step 54, are used instead of the current velocity vectors and poses passed from step 34. The pedestrians' destinations from step 34, however, are retained. With each iteration of sub-steps 51 to 54, another entry is created in the motion map 35 for a new time interval.The movement map 35 is finally passed to the control unit 40, so that it can read out the most probable locations of all persons in the prediction area for each time interval.

[0023] The sequence of sub-steps 51 to 53 is described in Fig. 3 This is further clarified. The movement of six people, numbers 61 to 66, is depicted there. The first three people, numbers 61 to 63, form a group. This group is clustered around a center point, 70. Within the predicted area 80, the members of the group are currently moving... Fig. 3 vertically upwards. The current velocity vectors v 64-66 akt< of the other pedestrians 64 to 66 are in the Fig. 3 Represented by arrows. Within the group, attractive social interactions F 61-63 att< act on the group members, causing them to move towards the center point 70. These attractive social interactions F 61-63 att< can be calculated for each pedestrian i according to Formula 1: F i att = α ⋅ U i ⋅ q

[0024] Here, α denotes the strength of a group attraction effect, and Ui is a unit vector pointing from pedestrian i to the center point 70. The value q is a threshold that indicates whether the attractive social interaction is effective at all. If the distance between pedestrian i and center point 70 is below the threshold, then Fiatt <= 0. The group attraction effect is therefore only active if pedestrian i moves further away from center point 70 than the threshold.

[0025] Further social interactions F 61-62 vis< cause the first two pedestrians 61, 62 of the group to reduce their speed so that the third pedestrian 63 does not lose contact with the group. These further social interactions F 61-62 vis< can be calculated for each pedestrian i according to Formula 2: F i vis = − β ⋅ v i akt ⋅ γ i

[0026] Here, β denotes the strength of an interaction within the group. The angle between the current velocity vector viact of pedestrian i and his direction of gaze is denoted by γi.

[0027] A further movement of all pedestrians 61 to 66 along their current velocity vectors is not possible without a collision. Therefore, repulsive social interactions F 61-66 soc< act, causing each of the pedestrians 61 to 66 to avoid the other pedestrians. These repulsive social interactions F i,j soc< can be calculated for each pedestrian i in relation to another pedestrian j according to Formula 3: F i , j soc = a j ⋅ e r i , j − d i , j b j ⋅ n i , j ⋅ λ + 1 − λ ⋅ 1 + cos φ i , j 2

[0028] Here, aj ≥ 0 denotes the strength and bj > 0 the direction of the repulsive social interaction. The distance between the two pedestrians i, j is denoted di,j, and ri,j denotes the sum of their radii. An anisotropy factor λ ∈ [0, 1] scales the repulsive social interaction in the direction of the movement of pedestrian i. The interaction reaches its maximum value when the angle φ i,j between a normalized vector ni,j, pointing from pedestrian i to pedestrian j, and the current velocity vector vi akt< of pedestrian i is zero. It is minimal when φ i,j = π.

[0029] Taking all these factors into account, future velocity vectors v61-66 of pedestrians 61 to 66 are determined, which differ from their current velocity vectors. The future movement of all pedestrians 61 to 66 is represented by dashed lines. It can be seen that the future velocity vectors v61-66 will change over the individual time intervals such that each pedestrian 61 to 66 will move back towards their original destination after passing the other pedestrians.

[0030] Fig. 4 shows how the current movement of pedestrians 61 to 63, who are members of the group, can be represented by a common current velocity vector v70, originating from the center 70 of the group. A self-driving vehicle 90, which is in the Fig. 4 In the depicted situation, the vehicle 90, which is planning a horizontal movement to the left, can detect pedestrians 61 to 63 by means of a sensor 91, which is implemented here as an RGB-D camera. Using the method according to the invention, its electronic control unit 92 predicts the movement of the group members in order to adjust its direction of movement, to reduce its speed if necessary, or to stop and thus avoid a collision with the group members. Other pedestrians, who themselves could not move into the direction of movement of the vehicle 90, are nevertheless detected by the sensor 91 and taken into account in the prediction, since they can influence the future movement of the group members through social interactions.

[0031] Based on the simulation environment of the ATC department store in Osaka, Japan, described in D. Brscic, T. Kanda, T. Ikeda, T. Miyashita, "Person position and body direction tracking in large public spaces using 3D range sensors", IEEE Transactions on Human-Machine Systems, Vol. 43, No. 6, pp. 522-534, 2013, a prediction according to the invention B1 and two predictions according to comparative examples VB1 and VB2 were performed. For comparative example VB1, a prediction method according to V. Karasev, A. Ayvaci, B. Heisele, S. Soatto, "Intent-aware long-term prediction of pedestrian motion", in 2016 IEEE International Conference on Robotics and Automation (ICRA), May 2016, was used. In this prediction method, a map 11 of the prediction range 80 is taken into account. However, the current speed vectors (v 61-66 akt< ) of other pedestrians 61-66 in the prediction area 80 are not taken into account in the prediction.For the comparative example VB2, a prediction method according to J. Elfring, R. Van De Molengraft, M. Steinbuch, "Learning intentions for improved human motion prediction", Robotics and Autonomous Systems, vol. 62, no. 4, pp. 591-602, 2014, was used. In this method, current velocity vectors v61-66 of other pedestrians 61-66 within prediction area 80 are taken into account. However, map 11 of prediction area 80 is not considered.

[0032] In the inventive example B1 and in the comparative examples VB1 and VB2, 21 scenarios with a total of 172 people were simulated, including 90 pedestrians in groups, with a total of 15 different possible destinations. A prediction was made over a period t of 12 seconds. The average Negative Log Probability (NLP) obtained in the respective simulations is shown in Fig. 6The average Modified Hausdorff Distance (MHD) obtained is shown in Fig. 7. It can be seen that the method according to the invention yields lower values ​​for the NLP and the MHD than the comparison examples VB1 and VB2. Consequently, the method according to the invention is more accurate.

Claims

1. Method for predicting (30) at least one future speed vector (v61-66zuk) and / or a future pose of a pedestrian (61-66) in a prediction region (80), characterized in that that a map (11) of the prediction region (80), through which the vehicle will move in the future and which contains all the geographical features of the prediction region (80), as created by sensors of a vehicle, and current speed vectors (v61-66akt) of further pedestrians (61-66) in the prediction region (80) are taken into account in the prediction, wherein the map (11) is used to create a movement plan (21), in which it is possible to see the sections of the prediction region (80), which is not blocked by obstacles, in which pedestrians (61-66) can move.

2. Method according to Claim 1, characterized in that, for the prediction (30), pedestrians (61-63) in the prediction region (80) are combined (53) into groups, wherein current poses and current speed vectors (v61-63akt) of all members of a group are used to determine a current pose and a current speed vector (v70akt) of the group.

3. Method according to Claim 1 or 2, characterized in that social interactions (F61-66soc, F61-63att, F61-62vis) between the pedestrians (61-66) are taken into account (52) in each case in the prediction (30).

4. Method according to one of Claims 1 to 3, characterized in that each pedestrian (61-66) is assigned (34) a destination and it is taken into account in the prediction (30) that this pedestrian is moving towards the destination.

5. Method according to one of Claims 1 to 4, characterized in that the prediction (30) is carried out in multiple temporal substeps, wherein, for each substep, all values taken into account in the prediction of the speed vectors (v61-66zuk) are recalculated.

6. Method according to Claim 5, characterized in that a future pose of the pedestrian (61-66) is determined in each substep and a movement map of the pedestrian (61-66) is created (35) from all determined future poses.

7. Method according to one of Claims 1 to 6, characterized in that current poses and current speed vectors (v61-66akt) of the pedestrians (61-66) are determined by means of at least one sensor (91) selected from the group consisting of monocular sensors, stereo sensors and depth sensors.

8. Computer program which is configured to carry out each step of the method according to one of Claims 1 to 7.

9. Computer program according to Claim 8, comprising a prediction module (20) with a movement plan (21), tracklets (22) and social context information (23) used for the prediction (30).

10. Machine-readable storage medium on which a computer program according to Claim 8 or 9 is stored.

11. Electronic control unit (93) which is configured to predict a future speed vector (v61-66zuk) and / or a future pose of a pedestrian (61-66) by means of a method according to one of Claims 1 to 7.