Method for predicting a movement of a pedestrian and driver assistance system

By using head and body bounding rectangles within a recurrent neural network architecture, the method stabilizes pedestrian trajectory predictions, addressing the instability in existing systems and improving the accuracy and safety of autonomous vehicle navigation.

WO2024240426A1PCT designated stage expired Publication Date: 2024-11-28MERCEDES BENZ GROUP AG
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/060632
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-25
Filing Date
2024-04-18
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing pedestrian movement prediction systems face instability and inaccuracy due to 'wobbling effects' caused by changes in bounding rectangle dimensions, particularly in egocentric camera views, which affect the reliability of predicting pedestrian trajectories.

Method used

The method involves generating and using both head and body bounding rectangles to stabilize the prediction model, employing a recurrent neural network with an encoder-decoder architecture to process temporal sequences of these rectangles, thereby reducing dimension fluctuations and improving trajectory prediction accuracy.

Benefits of technology

This approach leads to more stable and accurate predictions of pedestrian movements, enhancing the safety of autonomous vehicle operations by compensating for 'wobbling effects' and providing a robust driver assistance system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024060632_28112024_PF_FP_ABST
    Figure EP2024060632_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for predicting a movement of a pedestrian (10) in vehicle surroundings (3) of a vehicle (1), wherein at least one position and / or one movement of the pedestrian (10) is extracted from a plurality of recorded surroundings images (20.n), wherein during extraction of the current position and / or movement of the pedestrian (10) from the recorded surroundings images (20.n), at least a head bounding box (22.1) for a head (10.1) of the pedestrian (10) under consideration is generated as a first bounding box (22.1), and a body bounding box (22.2) for a body (10.2) of the pedestrian (10) is generated as a second bounding box (22.2), said bounding boxes being supplied to a prediction model (202) which predicts and outputs a trajectory (204) of the pedestrian (10) using the head bounding box (22.1) and body bounding box (22.2).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method for predicting a pedestrian's movement and driver assistance system

[0002] The invention relates to a method for predicting a movement of a pedestrian according to the preamble of claim 1, a computer program product and a driver assistance system.

[0003] Systems for predicting pedestrian movements are generally known. For example, DE 102019 134 048 A1 discloses a method for predicting pedestrian behavior for an ADAS application, comprising the steps of: capturing movement data (10) of a pedestrian (6) using vehicle sensors (3) of a vehicle (2); providing typical pedestrian movement patterns; recognizing the typical movement patterns in the movement data (10) by analyzing the movement data (10) using a neural network (4); categorizing the movement data (10) into categorized movement data (11) based on the recognized typical movement patterns; determining a prediction (12) of future behavior of the pedestrian (6) based on the categorized movement data (11) using a computing device (5).

[0004] DE 102021 117458 A1 discloses a method for operating a vehicle in which a risk level is calculated for a pedestrian about to cross a road, and a warning is issued to the pedestrian in response to the calculated risk level. This involves capturing images of the vehicle's surroundings using one or more cameras, generating so-called bounding boxes of the pedestrian in the images, and using the pedestrian's bounding boxes to predict whether the pedestrian will cross the road.Furthermore, it is intended that in addition to the bounding boxes of the pedestrian, bounding boxes of its head posture are also generated and that the bounding boxes of the head posture are used to predict whether the pedestrian performs a certain action, for example, writing a text message or making a phone call, or to determine whether the pedestrian is looking at the vehicle in order to estimate whether it perceives the vehicle.

[0005] US 2020 / 0410 224 A1 discloses a method for detecting pedestrians in an image of a vehicle's surroundings. The image is analyzed to identify a head in the vehicle's surroundings. Head detection is used to determine whether a pedestrian is present in the image. The image area containing the pedestrian is then enclosed in a bounding box, and based on this, a trajectory along which the vehicle should move is determined using a vehicle planning system.

[0006] From US 2014 / 0 347475 A1 a method for detecting pedestrians in video images is known, in which bounding boxes that completely frame the pedestrians or that partially frame the pedestrians in the case of temporary occlusion are generated in the video images and in which the bounding boxes are used to track the pedestrians in the video images.

[0007] From DE 102019209644 A1 a method for training a neural network based on an encoder-decoder architecture is known.

[0008] The invention is based on the object of providing a more robust method for predicting a pedestrian's movement as well as a correspondingly improved computer program product and a correspondingly improved driver assistance system.

[0009] The first-mentioned object is achieved according to the invention by a method for predicting a pedestrian's movement having the features of claim 1. The second-mentioned object is achieved according to the invention by a computer program product having the features of claim 9. The third-mentioned object is achieved according to the invention by a driver assistance system having the features of claim 10.

[0010] Advantageous embodiments of the invention are the subject of the subclaims.

[0011] In the method according to the invention for predicting a movement of a pedestrian in a vehicle environment of a vehicle, at least one position and / or one movement of the pedestrian is / are extracted from a plurality of recorded environmental images for a specific observation horizon, wherein when extracting the position(s) and / or the movement(s) of the pedestrian from the recorded environmental image(s), at least as a first bounding rectangle (also called a bounding box or bounding volume) a head bounding rectangle for a head of the pedestrian to be observed and as a second bounding rectangle a body bounding rectangle for a body of the pedestrian are generated, in particular position-related or movement-related, and fed to a prediction model which predicts and outputs a trajectory of the pedestrian on the basis of the at least one head bounding rectangle and the at least one body bounding rectangle.

[0012] By dividing a detected object, such as a pedestrian, into two or more bounding rectangles surrounding the object for object parts or sections, such as the pedestrian's head bounding rectangle and body bounding rectangle, the prediction module operates robustly and stably, as volume / rectangle changes and the resulting so-called "wobbling effects" can be reduced. This enables a more accurate prediction of the pedestrian's future trajectories in the image plane and safer behavior of an autonomous vehicle.

[0013] The invention is based on the idea that, during a pedestrian's movement, a body section in the area of ​​the head remains essentially unchanged with regard to its circumference or volume (also called the head boundary rectangle), whereas the body section in the area of ​​the legs or torso changes with regard to its circumference or volume (also called the body boundary rectangle) due to movements such as running, walking, jumping, or the like. If the prediction is limited only to the changing body section, which determines the intended movement and thus the prediction of the movement, such a method works more efficiently, robustly, and stably.

[0014] In particular, the invention can be used to reduce the so-called wobbling (= variability of the size / dimensions, in particular the width and / or height of the object to be viewed) of the bounding rectangle representing the object, which enables more stable and reliable predictions in connection with the determination of future pedestrian movements, in particular their direction and their path, in egocentric camera views.

[0015] A bounding rectangle is understood to be a simple geometric shape whose axes run parallel to the axes of the object and which encloses the object, such as a rectangle that encloses or surrounds a pedestrian. Instead of a bounding rectangle, a bounding box or a bounding volume can also be used in the method described here. To calculate the coordinates of the bounding rectangle or bounding box, the minimum and maximum x, y, and z coordinates of the object to be enclosed are determined. These are then used to determine all four corners of the bounding rectangle or all eight corners of the bounding box.

[0016] A recurrent neural network (RNN) is specifically understood as a feedback network that predicts or predicts a next probable scenario (= trajectory or output sequence of the used shapes) from temporal sequences of data and information (also called sequential features or input sequences) based on the shapes or patterns used (including simple geometric shapes such as rectangles, boxes, cuboids, or the like). A recurrent neural network comprises feedback loops to process a sequence of data or information, such as a sequence of position data or movement data.The recurrent neural network is a unidirectional network that considers the sequence of data or information, such as the sequence of temporally recorded position data or movement data, and considers and predicts the movement and a resulting probable scenario, a trajectory or output sequence, only in one direction forward into the future.

[0017] A prediction model (also called a forecast model) is understood to be a model for predicting one or more output variables, in particular, in this case, a pedestrian's movement probability based on a sequence of future bounding rectangles. The prediction model specifically predicts a pedestrian's movement probability based on linear relationships between model components, especially simple geometric shapes, and a behavioral probability.

[0018] A trajectory is understood, in particular, to be a movement path, in this case, that of a pedestrian. One possible embodiment provides that the prediction model is or will be configured to view the vehicle surroundings from a camera perspective of a moving vehicle based on the recorded environmental images and to predict or forecast the position and / or movement of the pedestrian in relation to image positions in the real environment.

[0019] The prediction model can also be configured to determine and compare changes in the head boundary rectangle and simultaneous changes in the body boundary rectangle over a period of time and to take them into account when predicting the pedestrian's movement.

[0020] A further development provides a pedestrian detector by means of which the head boundary rectangle and the body boundary rectangle are extracted from the captured environmental images.

[0021] The prediction model can, for example, be implemented as a recurrent neural network model with two recurrent neural networks in an encoder-decoder architecture. In other words, the neural network model comprises two recurrent neural networks, in particular a recurrent neural encoder network and a recurrent neural decoder network.

[0022] For example, the temporally recorded position information and / or movement information of the pedestrian is encoded into a temporal sequence of head bounding rectangles and body bounding rectangles by means of the recurrent neural encoder network and used as input for the recurrent neural decoder network in order to predict or predict future movements, in particular trajectories, of the pedestrian by means of the recurrent neural network model.

[0023] Furthermore, it can be provided that the prediction model is continuously updated or adapted. For example, the prediction model can be updated or adapted based on currently determined temporal sequences of boundary shapes, such as boundary rectangles and / or boundary cuboids, and / or reference shapes, such as reference boundary rectangles and / or reference boundary cuboids, and / or current states, such as vehicle parameters. For example, using such an updateable and adaptable prediction model, a braking or acceleration process initiated by the vehicle's own vehicle can be taken into account when predicting the pedestrian's trajectory.

[0024] Preferably, the surrounding image(s) are captured using at least one vehicle-mounted camera.

[0025] A further aspect of the invention is a computer program product which is configured to carry out all steps of the method described above.

[0026] A further aspect of the invention is a driver assistance system which is configured to carry out all steps of the method described above

[0027] Embodiments of the invention are explained in more detail below with reference to a drawing.

[0028] Showing:

[0029] Fig. 1 schematically shows a vehicle with a driver assistance system which is coupled to an environment detection unit and an output unit and which is configured to carry out a prediction method,

[0030] Fig. 2 schematically shows a temporal sequence of environmental images recorded by the environmental detection unit and bounding rectangles of variable width extracted therefrom, and

[0031] Fig. 3 schematically shows a flow diagram of a method according to the invention for predicting a movement of a pedestrian in a vehicle environment of a vehicle.

[0032] Corresponding parts are provided with the same reference numerals in all figures.

[0033] Figure 1 schematically shows a vehicle 1 with a driver assistance system 2, which is signal-coupled to an environment detection unit 4 for detecting a vehicle environment 3 and an output unit 6.

[0034] The driver assistance system 2 is configured to execute a prediction method 200, as shown in Figure 3. The driver assistance system 2 is, for example, a control unit or another computing unit in the vehicle 1. The driver assistance system 2 comprises a computer program product 5 (also referred to as application software), which is configured to execute the steps of the prediction method 200 (shown in Figure 3) based on a prediction model 202.

[0035] The environment detection unit 4 is, for example, a camera, in particular a 360° camera, a radar unit or the like, for detecting an environment 4.1 of the vehicle 1. The output unit 6 is, for example, a display unit in the vehicle, a control unit, another driver assistance module or the like.

[0036] The driver assistance system 2 can furthermore be coupled to sensors 8.n in terms of signals in order to determine, for example, a vehicle state and / or driving state, such as a driving speed, a driving acceleration, a braking operation or the like.

[0037] The environment detection unit 4, for example a camera, detects the vehicle environment 3 around the vehicle 1 and in particular objects located and / or moving in this vehicle environment 3, for example a pedestrian 10. The environment detection unit 4 is configured to capture a plurality of environment images 20.n and to feed them to the driver assistance system 2, in particular to the prediction model 202.

[0038] The prediction model 202 is configured to generate as output a prediction of pedestrian movements, in particular a trajectory 204 in egocentric camera views and to output it, for example, via the output unit 6.

[0039] An egocentric camera view is understood, in particular, to mean that the vehicle environment 3 is viewed from the camera perspective of a moving vehicle 1, and the behavior of the pedestrian(s) 10 is predicted with respect to the image positions in real traffic scenes. In other words, the pedestrian 10 is considered the center of the captured environmental images 20, and its movement in the real vehicle environment 3 is predicted. Such environmental detection is particularly suitable for autonomous driving of the vehicle 1.

[0040] The invention enables a prediction of pedestrian movements or pedestrian trajectories with compensation of wobble effects by being based on a more stable detection, coding and decoding of pedestrians 10 in captured environmental images 20.n by means of the prediction model 202 in the past.

[0041] Figure 2 schematically shows a temporal sequence 206 of environmental images 20.1 to 20.n recorded by the environmental detection unit 4 and body boundary rectangles 22.1 to 22.2 extracted therefrom with variable width 300 at different times tn. The body boundary rectangles 22.1 to 22.2 are generated for a pedestrian 10 detected in the environmental images 20.1 to 20.n and form a frame or box corresponding to the outline of the detected pedestrian 10.

[0042] For example, an intended pedestrian movement 208, in particular its planned path or trajectory, is represented as a sequence of several body boundary rectangles 22.1, 22.2 at the different times tn in the image plane.

[0043] Due to the movement of the pedestrian 10, for example short steps or long steps, fluctuations in the representation occur, in particular in the width 300 of the extracted bounding rectangles 22.1 to 22.2 (also called bounding box).

[0044] These fluctuations in width 300 are primarily due to the natural gait cycle of pedestrians 10. This effect is referred to as "box wobbling" and is illustrated in Figure 2. Neglecting this effect complicates the prediction of pedestrians' 10 movement patterns and can lead to poorer predictions.

[0045] To reduce this effect, the invention provides that when extracting the current position and / or movement of the pedestrian 10 from the recorded environmental images 20, n at least as a first bounding rectangle 22.1 a head bounding rectangle (hereinafter referred to as head bounding rectangle 22.1) for the head 10.1 of the pedestrian 10 to be observed and as a second bounding rectangle 22.2 a body bounding rectangle (hereinafter referred to as body bounding rectangle 22.2) for the body 10.2 of the pedestrian 10 are generated and fed to the prediction model 202, which predicts and outputs a corresponding trajectory 204 (shown in Figure 3) of the pedestrian 10 based on the head bounding rectangle 22.1 and the body bounding rectangle 22.2.

[0046] The invention is based on the idea of ​​incorporating information about the head position of pedestrians 10 into the prediction model 202 for predicting pedestrian paths or trajectories 204 of pedestrian 10, since separate head bounding rectangles 22.1 of the same head 10.1 of pedestrian 10 (separate bounding boxes for head 10.1) do not have the property of changing their shape at the different times tn, as shown in the sequence 206. This is in the nature of things, since the width of the same head 10.1 does not change, at least not perceptibly.

[0047] In contrast, the body bounding rectangles 22.2 (separate bounding boxes for the body 10.2) generated at different times tn and representing the body 10.2 of the same pedestrian 10 can vary in width 300 due to the gait cycle of this pedestrian 10, as shown in the sequence 206 and the associated width-time diagram with width 300 over time t. At time t3, the body bounding rectangle 22.2 of the detected pedestrian 10 has a smaller width than at the other times t1, t2, t4, and tn due to its movements.

[0048] The inclusion and consideration of shared information of pedestrian 10 by the static head boundary rectangles 22.1 and the varying body boundary rectangles 22.2 enables improved processing of this information by the prediction model 202, in particular, a more stable encoding of a movement history of pedestrian 10. Shaking effects of the boundary rectangles 22.n (shown in Figure 3) to be processed by the prediction model 202, the combination of static head boundary rectangles 22.1 and varying body boundary rectangles 22.2, can thus be easily reduced.

[0049] Figure 3 schematically shows a flow chart of the prediction method 200 for predicting the movement of the pedestrian 10 in the vehicle environment 3 of the vehicle 1.

[0050] In a first step 100, at least one position and / or one movement of the pedestrian 10 is or will be extracted from the environmental image 20.n. In particular, when extracting the current position and / or movement of the pedestrian 10 from the recorded environmental images 20.n, at least the head boundary rectangle 22.1 for the head 10.1 of the pedestrian 10 to be observed and the varying body boundary rectangle 22.2 for the body 10.2 of the pedestrian 10 are generated as body boundary rectangles 22.n and fed to the prediction model 202, which predicts the future trajectory 204 of the pedestrian 10 based on the static head boundary rectangle 22.1 and the varying body boundary rectangle 22.2 and outputs it in the real environmental image 20.n. The driver assistance system 2 comprises in particular a pedestrian detector by means of which the head boundary rectangle 22.1 and the body boundary rectangle 22.2 are extracted.

[0051] The prediction model 202 is configured to view the vehicle environment 3 from a camera perspective of a moving vehicle 1 based on the recorded environmental images 20.n and to predict the position and / or movement of the pedestrian 10 with respect to image positions in the real environment 3.

[0052] In addition, the prediction model 202 is configured to determine changes in the head boundary rectangle 22.1 and simultaneous changes in the body boundary rectangle 22.2 over a period of time t, to compare them with each other, and to take them into account when predicting the trajectory 204 of the pedestrian 10.

[0053] The prediction model 202 is implemented, for example, as a recurrent neural network model 210 with two recurrent neural networks in an encoder-decoder architecture. For example, the recurrent neural network model 210 comprises a recurrent neural encoder network 210.1 and a recurrent neural decoder network 210.2.

[0054] For example, the temporally recorded position information and / or movement information of the pedestrian 10, in particular the extracted boundary rectangles 22.n, such as the head boundary rectangles 22.1 and the body boundary rectangles 22.2, are encoded in a step 101 by means of the recurrent neural encoder network 210.1 into a temporal sequence 206 of head boundary rectangles 22.1 and body boundary rectangles 22.2 and used as input for the recurrent neural decoder network 210.2, which in step 102 generates and thus predicts or forecasts future movements, in particular trajectories 204, of the pedestrian 10 in the real environmental image 20.n. Subsequently, the trajectories 204 generated by the recurrent neural decoder network 210.2 in the real environment image 20.n are fed to the output unit 6 in step 103 and output there.

[0055] The prediction model 202 can be continuously updated or adapted. For example, the prediction model 202 can be updated or adapted based on currently determined temporal sequences 206 of boundary shapes, such as boundary rectangles 22 and / or boundary cuboids, and / or reference shapes, such as reference boundary rectangles and / or reference boundary cuboids, and / or current states, such as vehicle parameters. For example, using such an updateable and adaptable prediction model 202, a braking or acceleration process initiated by the host vehicle 1 can be taken into account when predicting the trajectory 204 of the pedestrian 10.

[0056] In other words, the prediction method 200 is a multi-stage method for predicting the future trajectories 204 or movement sequences of pedestrians 10 detected in the vehicle environment 3. First, the image positions of the pedestrians 10 are extracted from camera images of the moving vehicle 1 for a specific observation horizon. In contrast to other systems, the prediction method 200 generates, in particular extracts, and uses both body bounding boxes 22.2 and head bounding boxes 22.1 of the pedestrians 10 to determine their positions in the environmental image 20. The body and head bounding boxes (= head bounding rectangles 22.1 and body bounding rectangles 22.2) are extracted with the aid of a pedestrian detector. This information is then used as input to calculate the trajectories 204 of pedestrians 10 in the future environment images 20.n to predict.

[0057] The addition and use of these additional head boundary rectangles 22.1 as headbox information in the prediction method 200 leads to a more accurate prediction of the future trajectories 204 of pedestrians 10 in the image plane and enables safer behavior of an autonomous vehicle 1.

[0058] The advantages achieved with the invention consist in a more stable and robust representation of the pedestrian trajectories 204 in the image space of the environmental image 20.n as well as in a more accurate prediction of future pedestrian paths.

Claims

Patent claims 1. A method for predicting a movement of a pedestrian (10) in a vehicle environment (3) of a vehicle (1), wherein at least one position and / or one movement of the pedestrian (10) is extracted from a plurality of recorded environmental images (20.n), characterized in that when extracting the current position and / or movement of the pedestrian (10) from the recorded environmental images (20.n), at least as a first boundary rectangle (22.1) a head boundary rectangle (22.1) for a head (10.1) of the pedestrian (10) to be observed and as a second boundary rectangle (22.2) a body boundary rectangle (22.2) for a body (10.2) of the pedestrian (10) are generated and fed to a prediction model (202), which, based on the head boundary rectangle (22.1) and the body boundary rectangle (22.2), generates a trajectory (204) of the pedestrian (10) is predicted and output.

2. Method according to claim 1, characterized in that the prediction model (202) is or will be set up to view the vehicle environment (3) from a camera perspective of a moving vehicle (1) on the basis of the recorded environmental images (20. n) and to predict the position and / or the movement of the pedestrian (10) with respect to image positions in the real vehicle environment (3).

3. Method according to claim 1 or 2, characterized in that the prediction model (202) is or will be set up, Changes to the header boundary rectangle (22.1) and simultaneous changes to the Body boundary rectangle (22.2) over a period of time and to compare them with each other and to take them into account when predicting the trajectory (204) of the pedestrian (10).

4. Method according to one of the preceding claims, characterized in that a pedestrian detector is provided, by means of which the head boundary rectangle (22.1) and the body boundary rectangle (22.2) are extracted.

5. Method according to one of the preceding claims, characterized in that the prediction model (202) is designed as a recurrent neural network model (210) with two recurrent neural networks in an encoder-decoder architecture.

6. The method according to claim 5, characterized in that the temporally recorded position information and / or movement information of the pedestrian (10) is encoded by means of a recurrent neural encoder network (210.1) into a temporal sequence (206) of head boundary rectangles (22.1) and body boundary rectangles (22.2) and is used as input for a recurrent neural decoder network (210.2) in order to predict the trajectories (204) of the pedestrian (10) by means of the recurrent neural network model (210).

7. Method according to one of the preceding claims, characterized in that the prediction model (202) is updated or adapted on the basis of currently determined temporal sequences (206) of boundary shapes, reference shapes and / or current states.

8. Method according to one of the preceding claims, characterized in that the surrounding images are captured by means of at least one vehicle-mounted camera.

9. Computer program product (5) which is configured to carry out all steps (100-103) of a prediction method (200) according to one of the preceding claims.

10. Driver assistance system (2) which is configured to carry out all steps (100-103) of a prediction method (200) according to one of the preceding claims 1 to 8.

Citation Information

Patent Citations

  • Methods for predicting pedestrian behavior

    DE102019134048A1

  • Methods for training a neural network

    DE102019209644A1

  • Method for determining the quantity of carbon dioxide emitted in relation to the goods during a goods transport and tracking device for its implementation

    DE102021117458A1

  • Real-time object detection, tracking and occlusion reasoning

    US20140347475A1

  • Head Detection for Improved Pedestrian Detection

    US20200410224A1