Method for predicting the trajectory of an object

Abstract

The present invention relates to a method for predicting the trajectory of an object around a vehicle, in which sensor raw data (SR) around the vehicle is sensed by an environmental sensor and pre-processed at multiple consecutive time steps (t_0, t_(-1), t_(-T)) to generate object hypotheses (OH) for the objects around the vehicle, the sensor raw data (SR) is segmented and assigned to each object hypothesis (OH) based on the object hypotheses (OH), the sensor raw data (SR) belonging to each object hypothesis (OH) is converted into a latent encoding (LE) and assigned to each object hypothesis (OH), and the sensor raw data (SR) belonging to each object hypothesis (OH) is converted into a latent encoding (LE) and assigned to each object hypothesis (OH). Features are assigned to the object hypotheses (OH), and a fused object hypothesis (FOH) is created by learning-based clustering from the individual object hypotheses (OH) and the assigned features. A track (T) of each fused object hypothesis (FOH) is formed by creating a learning-based assignment between the fused object hypothesis (FOH) calculated at the current time step (t_0) and the fused object hypotheses (FOH) calculated at multiple past time steps (t_(-1), t_(-T)). A trajectory (PT) is predicted based on the track (T) of each fused object hypothesis (FOH).

Description

[Technical Field]

[0001] The present invention relates to a method for predicting the trajectories of objects in the vicinity of a vehicle. [Background technology]

[0002] Prediction of vehicles participating in traffic is a key component of autonomous vehicles: reliable and safe trip planning is only possible if this prediction is also of high quality.

[0003] From DE 10 2019 215 147 A1 a method is known for guiding an own vehicle in the following steps: - detecting an external surrounding environment of the vehicle and outputting surrounding environment information from the detected surrounding environment; - predicting the trajectories of road users around the vehicle based on the output surrounding environment information using a neural network; - predicting the trajectories of road users around the vehicle based on the output surrounding environment information; - calculating the collision risk between the vehicle and surrounding road users for each of the neural network-based and rule-based predicted trajectories; - selecting a neural network-based or rule-based predicted trajectory for each road user depending on the calculated collision risk; - providing an ego-trajectory for guiding the ego-vehicle according to a selected predicted trajectory of a road user; is.

[0004] Furthermore, from DE 10 2019 216 290 A1, a method is known for tracking an object depending on sensor data from an environment sensor for vehicle operation, where the vehicle has an environment sensor and the sensor data represent the environment around the vehicle as detected by the environment sensor. The method comprises the following steps: - selecting a subset of sensor data depending on the current state of the tracked object; - applying machine learning methods to a selected subset of the sensor data to obtain information about the tracked objects from the data; - updating the current state of the tracked object depending on the information obtained; is. Summary of the Invention [Problem to be solved by the invention]

[0005] The object of the present invention is to provide a new method for predicting the trajectories of objects around a vehicle. [Means for solving the problem]

[0006] This object is achieved according to the invention by a method having the features set forth in claim 1.

[0007] Advantageous embodiments of the invention are the subject matter of the dependent claims.

[0008] In the method according to the present invention for predicting the trajectory of an object around a vehicle, sensor raw data around the vehicle is detected by an ambient environment sensor, the sensor raw data is pre-processed at multiple consecutive time steps to create object hypotheses for objects around the vehicle, the sensor raw data is segmented based on the object hypotheses and assigned to each object hypothesis, the sensor raw data belonging to each object hypothesis is converted into latent encoding by a learning-based encoder block and assigned as features to each object hypothesis, a fused object hypothesis is created in a fusion block using learning-based clustering from the individual object hypotheses and the assigned features, tracks of each fused object hypothesis are formed in a tracking block by creating a learning-based assignment between the fused object hypothesis calculated at the current time step and the fused object hypotheses calculated at multiple past time steps, and a trajectory is predicted based on the tracks of each fused object hypothesis.

[0009] Advantageously, raw sensor data is sensed by multiple environmental sensors of multiple sensor modalities and pre-processed separately for each sensor, and latent encodings are identified separately for each sensor.

[0010] In one embodiment, a prediction error is calculated by comparing the predicted trajectory of the fused object hypothesis for a future time point with the true trajectory of the fused object hypothesis calculated at the future time point, and the calculated prediction error is back-propagated to the encoder block, fusion block, and tracking block for training purposes.

[0011] In one embodiment, two or more of the following group are used as sensor modalities: camera, radar sensor, lidar sensor, and ultrasonic sensor.

[0012] In one embodiment, a Transformer model, a recurrent neural network, or a graph neural network is used as the algorithm for predicting the trajectory.

[0013] In one embodiment, the segmented raw sensor data of the camera object hypotheses is converted into a latent encoding using a convolutional neural network, and weights are learned in the convolutional neural network.

[0014] In one embodiment, the segmented raw sensor data of the lidar sensor object hypotheses is converted to a latent encoding using PointNet, and weights are learned in PointNet.

[0015] In one embodiment, to form a learning-based fused object hypothesis, pairwise membership criteria between nodes in the graph are calculated, a graph neural network is used for "link prediction" and / or edge classification to generate pairwise probabilities that nodes belong to the same object, and clusters of individual nodes are formed based on the membership criteria using standard clustering algorithms.

[0016] In an alternative embodiment of the method, a learned graph clustering algorithm is used to form the fused object hypotheses in a learned manner.

[0017] In one embodiment, for each cluster, the information of all nodes is aggregated using pooling, resulting in an aggregated latent representation of the sensor data and an aggregated state for each fused object hypothesis.

[0018] In one embodiment, graph neural networks are used for "link prediction" and / or edge classification for tracking.

[0019] Learning-based trajectory prediction methods have proven to be particularly accurate. This invention demonstrates for the first time how fusion, tracking, and prediction can be performed in an end-to-end learning approach. The learned end-to-end approach ensures that the relevant sensor information of each object hypothesis can also be used for prediction.

[0020] In known trajectory prediction methods that operate on tracklets (temporal sequences of two-dimensional x-y coordinates of individual vehicles in a scene), the tracklets come from an upstream stack that has already performed perception, tracking, and fusion of individual agents. A drawback of these approaches is that only tracklets are used as input information for prediction. Unlike these approaches, our solution does not lose sensor-specific information (such as the color or shape of the detected vehicle). This can therefore be advantageous, as shape may be relevant for prediction. For example, a sports car may behave kinematically differently from a typical family car, and the presence of such information can also be advantageous for prediction.

[0021] In other trajectory prediction methods that operate on raw sensor data from a single sensor modality, object detection and prediction are learned end-to-end. The problem here is that these approaches are always limited to a single sensor modality: objects are detected almost exclusively using a lidar scanner, then tracked over time, and predicted. Unlike these approaches, our solution takes multiple sensor modalities (camera, lidar, radar, ultrasonic) into account, thereby fulfilling a key requirement of autonomous systems. All of these sensor modalities generate important information that can be used simultaneously by our solution.

[0022] Our approach allows us to utilize detections from any number of independent sensor modalities, fuse these detections, track them over time, and subsequently generate predictions, where the end-to-end learning-based approach allows relevant sensor information (which information is relevant for the prediction and how to extract it) to also be used for the prediction.

[0023] The approach according to the present invention allows for improved trajectory prediction by using relevant sensor information in the form of latent encoding, where which information is relevant is learned and not determined by manually created metrics. Improved prediction allows for better planning of the autonomous vehicle's behavior, which increases driving comfort and safety. The end-to-end approach avoids training individual components and allows for training as a whole, which saves training time. It is possible to use objects detected with different sensor modalities without any problems. Furthermore, scaling with any number of sensors and any sensor modalities is also possible.

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects around a vehicle; [Figure 2] 1 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects around a vehicle; [Figure 3] 1 is a schematic block diagram of a system for predicting the trajectories of objects around a vehicle; DETAILED DESCRIPTION OF THE INVENTION

[0026] In all the drawings, the same reference numerals are used to designate corresponding parts.

[0027] The present invention relates to a method for predicting the trajectory of an object around a vehicle. The vehicle has multiple sensors for detecting the surroundings, such as at least one camera, at least one radar sensor, at least one lidar sensor, and / or at least one ultrasonic sensor. The present invention assumes that raw sensor data from these sensors is preprocessed. This preprocessing is performed separately for each individual sensor (sensor). During the preprocessing, an object hypothesis is created. The object hypothesis is a data set extracted from the raw sensor data and containing information about the object. Such information is, for example, information about the type of object (pedestrian, vehicle) and the state of the object (position of the object in a coordinate system common to all sensors, size of the object). Part of the object hypothesis is a state vector and the raw sensor data.

[0028] For example, from data detected by a camera, an object hypothesis is calculated (determined) including images of detected objects and their respective positions in a coordinate system. For example, from data detected by a radar sensor, an object hypothesis is calculated including reflection points, positions, and velocities (radar also has the ability to measure velocity through the Doppler effect) of detected objects in a coordinate system.

[0029] An object hypothesis accordingly has, on the one hand, a state vector (hereinafter referred to as "state") containing information about the object hypothesis. This state comprises at least the position and size of the object hypothesis in a unified coordinate system (the position and size of the object for which the object hypothesis is created). Accordingly, other sensor-specific values ​​may also be part of the state of the object hypothesis. A radar detection may, for example, also comprise the speed. Furthermore, based on the object hypotheses (and their values), the raw sensor data of the object is segmented and assigned to the respective object hypothesis. For example, in the case of a camera or lidar sensor, the pixels of the detected vehicle may be extracted (semantic extraction of pixels of the detected vehicle).

[0030] This preprocessing is repeated for successive time steps.

[0031] The raw sensor data belonging to each object hypothesis is converted into a latent encoding by a learning-based encoder and assigned as features to each object hypothesis.

[0032] From the individual object hypotheses and assigned features, a fused object hypothesis is created by (learning-based) clustering.

[0033] In the next step, a track for each fused object hypothesis is formed by making a (learning-based) assignment between the fused object hypothesis calculated at the current time step and the fused object hypotheses calculated at multiple previous time steps.

[0034] In a further step, a trajectory is predicted based on the tracks of each fused object hypothesis. Possible algorithms for trajectory prediction include, for example, Transformer, RNN, and GNN.

[0035] FIG. 1 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects around a vehicle.

[0036] In the encoder block 1, a latent encoding LE is formed from the object hypotheses OH calculated in the preprocessing and the assigned raw sensor data SR. Here, a latent encoding LE is formed for each object hypothesis OH detected in the current time step and assigned as a feature to each object hypothesis OH. The latent encoding LE is a value from a preset restricted set of values. The raw sensor data SR is a value from an unrestricted set of values. That is, the encoding maps the raw sensor data SR from an unrestricted set of values ​​to a value from a restricted set of values. For camera object hypotheses OH, the learning-based encoder block 1 may be designed, for example, as follows: The segmented raw sensor data SR of the camera object hypotheses OH can be converted into a latent encoding LE using, for example, a convolutional neural network (CNN). In this case, weights are learned in the CNN. For lidar sensor object hypotheses OH, the learning-based encoder block 1 may be designed, for example, as follows: The segmented raw sensor data SR of the lidar sensor object hypotheses OH can be converted into a latent encoding LE using, for example, PointNet. In this case, weights are learned in PointNet.

[0037] That is, for each object hypothesis OH at each time step, we end up with a state and a learned latent encoding LE. The content of this latent encoding LE is not human-interpretable; it is a maximally appropriate representation of the raw sensor data SR that the model itself learned during training.

[0038] In fusion block 2, the object hypotheses OH of all sensors formed in the current time step are clustered by their assigned latent encodings LE to form a fused object hypothesis FOH. At this time, a graph is created for each time step. In this graph, all object hypotheses OH of the time step are nodes. Therefore, each node has a state vector and a latent encoding LE of the sensor data, which contains the learned appropriate representation. In the graph, all nodes are interconnected. Therefore, this is a fully connected graph.

[0039] The fused object hypothesis FOH can be formed by learning-based clustering in a graph. Here, the following two variations can be used: - Compute pairwise membership criteria between nodes in the graph. This membership criterion is learned. As with the learning-based encoder block 1, the error criterion required for this is determined only after the actual trajectory prediction and then backpropagated to determine the membership criterion. Based on the graph structure, for example, graph neural networks can be used for "link prediction" and / or "edge classification", which results in pairwise probabilities that nodes belong to the same object. Based on the membership criterion, clusters of individual nodes can be formed using standard clustering algorithms. Alternatively, it is also possible to use the learned graph clustering algorithm directly. In this alternative variation, the error criterion required for this is determined only after the actual trajectory prediction and then back-propagated to the determination of the membership criterion.

[0040] Thus, during the training process, the described learning-based clustering learns to assign object hypotheses OH to corresponding clusters such that the error in trajectory prediction is minimized. This happens when object hypotheses OH from multiple sensor modalities (e.g., camera and lidar) belonging to the same real object are also assigned to the same cluster.

[0041] In each cluster, the information of all nodes is aggregated (e.g., pooled). This corresponds to fusing multiple object hypotheses OH into one fused object hypothesis FOH. As a result, for each fused object hypothesis FOH, an aggregated latent representation of the sensor data and an aggregated state are obtained. The type of aggregation for this state can be, for example, averaging.

[0042] In the tracking block 3, the fused object hypothesis FOH of the current time step and the fused object hypothesis FOH calculated in previous time steps are analyzed. Over multiple time steps, it is calculated which fused object hypothesis FOH of previous time steps belongs to which fused object hypothesis FOH of the current time step. The related fused object hypothesis FOH from different time steps form a track T for each fused object hypothesis FOH. Track T represents the time progression of each fused object hypothesis FOH. For tracking, a graph can be constructed that includes all fused object hypothesis FOH of previous time steps as nodes and all fused object hypothesis FOH of the current time step as nodes. The feature vector of each node is again the latent encoding LE and the state. In the graph, all nodes of two consecutive time steps are interconnected by edges. Membership criteria are calculated only for nodes connected by edges. For this purpose, a graph neural network (GNN) can again be used for "link prediction" and / or "edge classification." For each node of the current time step, the node of the previous time step with the highest membership criterion can be identified. These nodes belong to the same object track T. As a result, a track T of fused object hypotheses FOH is obtained. That is, the fused object hypotheses FOH are assigned to each other over multiple time steps, thereby generating tracklets. Thus, at each time step, a track T is created through each fused object hypothesis FOH, to which the states of each fused object hypothesis FOH and their latent feature vectors are also assigned.

[0043] In prediction block 4, the trajectories PT of the fused object hypotheses FOH are predicted for future time steps based on their tracks T. This results in predicted trajectories PT or tracks of the various fused object hypotheses FOH.

[0044] The encoding in the encoder block 1, the clustering in the fusion block 2, and the membership formation in the tracking block 3 are performed by a learning algorithm. For training, the trajectory PT of the fused object hypothesis FOH at a future time point is predicted, and the prediction is compared with the true trajectory FT of the fused object hypothesis FOH calculated at that future time point to calculate a prediction error PE. The calculated prediction error PE is back-propagated to the encoder block 1, the fusion block 2, and the tracking block 3 for training the algorithm. Therefore, the algorithms in the encoder block 1, the fusion block 2, and the tracking block 3 are optimized end-to-end simultaneously.

[0045] Through Latent Encoding (LE), the trajectory prediction algorithm automatically accesses relevant sensor information propagated through the network.

[0046] 2 is a schematic diagram illustrating the steps of the method for predicting the trajectories of objects in the vicinity of the vehicle by backpropagation of the prediction error PE described above. As already mentioned, the prediction error PE is calculated by comparing the predicted trajectory PT with the true trajectory FT. In the diagram, this comparison is indicated by a circle.

[0047] FIG. 3 is a schematic block diagram of a system for predicting the trajectory PT of an object around a vehicle.

[0048] As inputs, object hypotheses OH1, OH2, OH3, OHm of different sensors are available, which may be of the same or different sensor modalities, e.g., camera, lidar, radar, and / or ultrasound. In each learning-based encoder block 1, a latent encoding LE is formed for the current time step t_0 from the object hypotheses OH1 to OHm and the assigned raw sensor data SR. In this case, one and the same encoder block 1 can be used for object hypotheses OH1 to OHm of the same sensor modality, with separate weighting if necessary.

[0049] In fusion block 2, the object hypotheses OH of all sensors formed at the current time step t_0 are clustered based on their respective assigned latent encodings LE to form a fused object hypothesis FOH.

[0050] In the tracking block 3, the fused object hypothesis FOH at the current time step t_0 and the fused object hypothesis FOH calculated at the past time steps t_(-1) and t_(-T) are analyzed. The mutually related fused object hypothesis FOH from different time steps t_0, t_(-1), and t_(-T) form a track T of each fused object hypothesis FOH.

[0051] In prediction block 4, the trajectories PT of the fused object hypotheses FOH are predicted based on their tracks T for future time steps. [Prior art documents] [Patent documents]

[0052] [Patent Document 1] German Patent Application Publication No. 102019215147 [Patent Document 2] German Patent Application Publication No. 102019216290

Claims

1. A method for predicting the trajectory (PT) of an object around a vehicle, comprising: the vehicle has a plurality of ambient environment sensors of a plurality of sensor modalities and a processor; In each of the ambient environment sensors, - raw sensor data (SR) of the surroundings of the vehicle is sensed, - the raw sensor data (SR) is processed at a number of successive time steps (t_0, t_(-1), t_(-T)) to generate an object hypothesis (OH), which is a data set containing information extracted from the raw sensor data (SR) and related to the object; - based on the object hypotheses (OH), a segmentation of the raw sensor data (SR) is performed to extract the objects, and each of the segmented raw sensor data (SR) is assigned to a respective object hypothesis (OH); In the processor, - each of the segmented raw sensor data (SR) belonging to each of the object hypotheses (OH) is converted by a learning-based encoder block (1) into a latent encoding (LE) which is a compressed representation of the features contained in the raw sensor data (SR), and the latent encoding (LE) is assigned as a feature to each of the object hypotheses (OH); - A fusion block (2) performs learning-based clustering on the object hypotheses (OH) created by each of the surrounding environment sensors based on the features assigned to the object hypotheses (OH) by the encoder block (1), thereby creating a fused object hypothesis (FOH) in which multiple object hypotheses (OH) are fused; - a track (T) of each of the fused object hypotheses (FOH) is formed in a tracking block (3) by creating a learning-based assignment between the fused object hypotheses (FOH) calculated at the current time step (t_0) and the fused object hypotheses (FOH) calculated at a number of previous time steps (t_(-1), t_(-T)), - based on the track (T) of each of the fused object hypotheses (FOH), the trajectory (PT) is predicted A method characterized by:

2. A trajectory (PT) of the fused object hypothesis (FOH) predicted for a future time point is compared with a true trajectory (FT) of the fused object hypothesis (FOH) calculated at the future time point to calculate a prediction error (PE), and the calculated prediction error (PE) is back-propagated to the encoder block (1), the fusion block (2), and the tracking block (3) for training purposes.

2. The method of claim 1.

3. The algorithm used to predict the trajectory (PT) is a transformer model, a recurrent neural network, or a graph neural network.

3. The method according to claim 1 or 2.

4. The surrounding environment sensor includes a camera, and the segmented sensor raw data (SR) of the object hypothesis (OH) of the camera is converted to the latent encoding (LE) using a convolutional neural network, and weights are learned in the convolutional neural network.

3. The method according to claim 1 or 2.

5. The ambient environment sensor includes a lidar sensor, and the segmented sensor raw data (SR) of the object hypothesis (OH) of the lidar sensor is converted to the latent encoding (LE) using PointNet, and weights are learned in the PointNet.

3. The method according to claim 1 or 2.

6. To form the Fused Object Hypothesis (FOH) in a learning-based manner, pairwise membership criteria are calculated between nodes in a graph representing all of the Object Hypothesis (OH) for the plurality of consecutive time steps, a graph neural network is used for "link prediction" and / or edge classification to generate pairwise probabilities that nodes belong to the same object, and clusters of the individual nodes are formed based on the membership criteria using standard clustering algorithms.

3. The method according to claim 1 or 2.

7. A trained graph clustering algorithm is used to form the Fused Object Hypothesis (FOH) based on learning.

3. The method according to claim 1 or 2.

8. For each cluster, information of all nodes representing all the object hypotheses (OH) for the plurality of consecutive time steps is aggregated using pooling, resulting in an aggregated latent representation of the raw sensor data (SR) and an aggregated state for each fused object hypothesis (FOH).

3. The method according to claim 1 or 2.

9. For the processing in the tracking block (3), a graph neural network is used for the "link prediction" and / or the edge classification.

7. The method according to claim 6.

Images