Method for predicting the trajectory of an object

Abstract

The present invention relates to a method for predicting the trajectory of an object around a vehicle, in which sensor raw data (SR) around the vehicle is sensed by an environmental sensor and pre-processed at multiple consecutive time steps (t_0, t_(-1), t_(-T)) to generate object hypotheses (OH) for the objects around the vehicle, the sensor raw data (SR) is segmented and assigned to each object hypothesis (OH) based on the object hypotheses (OH), the sensor raw data (SR) belonging to each object hypothesis (OH) is converted into a latent encoding (LE) and assigned to each object hypothesis (OH), and the sensor raw data (SR) belonging to each object hypothesis (OH) is converted into a latent encoding (LE) and assigned to each object hypothesis (OH). Features are assigned to the object hypotheses (OH), and a fused object hypothesis (FOH) is created by learning-based clustering from the individual object hypotheses (OH) and the assigned features. A track (T) of each fused object hypothesis (FOH) is formed by creating a learning-based assignment between the fused object hypothesis (FOH) calculated at the current time step (t_0) and the fused object hypotheses (FOH) calculated at multiple past time steps (t_(-1), t_(-T)). A trajectory (PT) is predicted based on the track (T) of each fused object hypothesis (FOH).

Description

[Technical field]

[0001] The present invention relates to a method for predicting the trajectories of objects in the vicinity of a vehicle. [Background technology]

[0002] Prediction of vehicles participating in traffic is a key component of autonomous vehicles: reliable and safe trip planning is only possible if this prediction is also of high quality.

[0003] From DE 10 2019215147 A1 a method is known for guiding an own vehicle in the following steps: - detecting an external surrounding environment of the vehicle and outputting surrounding environment information based on the detected surrounding environment; - predicting the trajectory of road users around the vehicle based on the output surrounding environment information on the basis of a neural network; - predicting the trajectory of road users around the vehicle based on the output surrounding environment information; - calculating a collision risk between the vehicle and surrounding road users for each of the neural network-based and rule-based predicted trajectories; - selecting a neural network-based or rule-based predicted trajectory for each road user depending on the calculated collision risk; - providing an ego-trajectory for guiding the ego-vehicle according to a selected foreseen trajectory of a road user; It is.

[0004] Furthermore, from DE 10 2019 216 290 A1 a method is known for tracking objects as a function of sensor data from an environment sensor for vehicle operation, where the vehicle has an environment sensor and the sensor data represents the environment around the vehicle detected by the environment sensor. The method comprises the following steps: - selecting a subset of the sensor data depending on the current state of the tracked object; - applying machine learning methods to a selected subset of the sensor data to obtain information about the tracked objects from the data; - updating the current state of the tracked object depending on the obtained information; It is. Summary of the Invention [Problem to be solved by the invention]

[0005] The object of the present invention is to provide a new method for predicting the trajectories of objects around a vehicle. [Means for solving the problem]

[0006] This object is achieved according to the invention by a method having the features set forth in claim 1.

[0007] Advantageous embodiments of the invention are the subject matter of the dependent claims.

[0008] In the method according to the present invention for predicting the trajectory of an object around a vehicle, sensor raw data around the vehicle is detected by an environmental sensor, the sensor raw data is pre-processed at multiple successive time steps to create object hypotheses for objects around the vehicle, the sensor raw data is segmented and assigned to each object hypothesis based on the object hypotheses, the sensor raw data belonging to each object hypothesis is converted to a latent encoding by a learning-based encoder block and assigned as features to each object hypothesis, a fused object hypothesis is created by learning-based clustering in a fusion block from the individual object hypotheses and the assigned features, tracks of each fused object hypothesis are formed in a tracking block by creating a learning-based assignment between the fused object hypothesis calculated at the current time step and the fused object hypotheses calculated at multiple past time steps, and a trajectory is predicted based on the tracks of each fused object hypothesis.

[0009] Advantageously, raw sensor data is sensed by multiple environmental sensors of multiple sensor modalities and pre-processed separately for each sensor, and latent encodings are identified separately for each sensor.

[0010] In one embodiment, a prediction error is calculated by comparing the trajectory of the fused object hypothesis predicted for a future time point with the true trajectory of the fused object hypothesis calculated at the future time point, and the calculated prediction error is back-propagated to the encoder block, the fusion block, and the tracking block for training purposes.

[0011] In one embodiment, two or more of the following group are used as sensor modalities: camera, radar sensor, lidar sensor, and ultrasonic sensor.

[0012] In one embodiment, a Transformer model, a recurrent neural network, or a graph neural network is used as an algorithm to predict the trajectory.

[0013] In one embodiment, the segmented raw sensor data of the camera object hypotheses is converted to a latent encoding using a convolutional neural network, and weights are learned in the convolutional neural network.

[0014] In one embodiment, the segmented raw sensor data of the lidar sensor object hypotheses is converted to a latent encoding using PointNet, and weights are learned in PointNet.

[0015] In one embodiment, pair-wise membership criteria between nodes in the graph are computed for learning-based formation of fused object hypotheses, a graph neural network is used for "link prediction" and / or edge classification to generate pair-wise probabilities that nodes belong to the same object, and clusters of individual nodes are formed based on the membership criteria using standard clustering algorithms.

[0016] In an alternative embodiment of the method, a learned graph clustering algorithm is used for learning-based formation of fused object hypotheses.

[0017] In one embodiment, for each cluster, the information of all nodes is aggregated using pooling, resulting in an aggregated latent representation of the sensor data and an aggregated state for each fused object hypothesis.

[0018] In one embodiment, graph neural networks are used for "link prediction" and / or edge classification for tracking.

[0019] Learning-based trajectory prediction methods have proven to be particularly accurate. We show for the first time how fusion, tracking and prediction can be performed in an end-to-end learning approach. The end-to-end learned approach ensures that the relevant sensor information of each object hypothesis can also be used for prediction.

[0020] In the case of known trajectory prediction methods that operate on tracklets (temporal sequences of 2D xy coordinates of individual vehicles in a scene), the tracklets come from an upstream stack that has already perceived, tracked and fused individual agents. A drawback of these approaches is that only the tracklets are used as input information for the prediction. Unlike these approaches, with the inventive solution, sensor-specific information (such as the color or shape of the detected vehicle) is not lost. This can therefore be advantageous, since the shape may be relevant for the prediction. For example, a sports car may behave kinematically differently than a typical family car, so the presence of such information can also be advantageous in the prediction.

[0021] In other trajectory prediction methods that operate on raw sensor data of a single sensor modality, object detection and prediction are learned end-to-end. The problem here is that these approaches are always limited to a single sensor modality: objects are detected almost exclusively with a lidar scanner, then tracked and predicted over time. Unlike these approaches, our solution takes into account multiple sensor modalities (camera, lidar, radar, ultrasonic), thereby fulfilling a key requirement of autonomous systems. All of these sensor modalities generate important information that can be used simultaneously by our solution.

[0022] Our approach allows for the use of detections from any number of independent sensor modalities, fusing these detections, tracking them over time, and subsequently generating predictions, where the end-to-end learning-based approach allows for relevant sensor information to also be used for prediction (learning which information is relevant for the prediction and how to extract it).

[0023] The approach according to the invention allows for improved trajectory prediction by using relevant sensor information in the form of latent encoding, where which information is relevant is learned and not specified by manually created metrics. The improved prediction allows for better planning of the behavior of the autonomous vehicle, which increases driving comfort and safety. In an end-to-end approach, the training of individual components is omitted and they can be trained as a whole, which saves training time. It is possible without problems to use objects detected with different sensor modalities. Furthermore, scaling with any number of sensors and any sensor modalities is also possible.

[0024] In the following, an embodiment of the present invention will be explained in detail with reference to the drawings. [Brief description of the drawings]

[0025] [Figure 1] FIG. 2 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects around a vehicle. [Diagram 2] FIG. 2 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects around a vehicle. [Diagram 3] 1 is a schematic block diagram of a system for predicting the trajectories of objects around a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] In all the drawings, the same reference numerals are used to designate corresponding parts.

[0027] The present invention relates to a method for predicting the trajectory of an object in the vicinity of a vehicle. The vehicle has several sensors for sensing the surroundings, for example at least one camera, at least one radar sensor, at least one lidar sensor and / or at least one ultrasonic sensor. The present invention presupposes that raw sensor data from these sensors is preprocessed. This preprocessing is performed separately for each individual sensor. During the preprocessing, an object hypothesis is created. An object hypothesis is a data set containing information extracted from the raw sensor data and information about the object. Such information is for example information about the type of object (pedestrian, vehicle) and the state of the object (position of the object in a coordinate system common to all sensors, size of the object). Part of the object hypothesis is a state vector and the raw sensor data.

[0028] For example, from data detected by a camera, an object hypothesis is calculated (determined) that includes images of detected objects and their respective positions in a coordinate system. For example, from data detected by a radar sensor, an object hypothesis is calculated that includes reflection points, positions, and velocities (radar also has the ability to measure velocities by the Doppler effect) of detected objects in a coordinate system.

[0029] An object hypothesis accordingly has on the one hand a state vector (hereinafter referred to as state, English "State") that contains information about the object hypothesis. This state comprises at least the position and size of the object hypothesis in a unified coordinate system (position and size of the object for which the object hypothesis is created). Correspondingly, other sensor-specific values ​​can be part of the state of the object hypothesis. Radar detections can for example also comprise the speed. Furthermore, based on the object hypotheses (and their values), the sensor raw data of the object is segmented and assigned to the respective object hypothesis. For example, in case of a camera or lidar sensor, the pixels of the detected vehicle will be extracted (semantic extraction of pixels of detected vehicles).

[0030] This preprocessing is repeated for successive time steps.

[0031] The raw sensor data belonging to each object hypothesis are converted into a latent encoding by a learning-based encoder and assigned as features to each object hypothesis.

[0032] From the individual object hypotheses and the assigned features, a fused object hypothesis is created by (learning-based) clustering.

[0033] In the next step, a track for each fused object hypothesis is formed by making a (learning-based) assignment between the fused object hypothesis calculated at the current time step and the fused object hypotheses calculated at multiple previous time steps.

[0034] In a further step, a trajectory is predicted based on the tracks of each fused object hypothesis. Possible algorithms for trajectory prediction include, for example, Transformer, RNN, and GNN.

[0035] FIG. 1 is a schematic diagram illustrating the steps of a method for predicting the trajectories of objects in the vicinity of a vehicle.

[0036] In the encoder block 1, a latent encoding LE is formed from the object hypothesis OH calculated in the preprocessing and the assigned sensor raw data SR. At this time, a latent encoding LE is formed for each object hypothesis OH detected in the current time step and assigned as a feature to each object hypothesis OH. The latent encoding LE is a value from a preset restricted value set. The sensor raw data SR is a value from an unrestricted value set. That is, the encoding maps the sensor raw data SR from an unrestricted value set to a value from a restricted value set. For the camera object hypothesis OH, the learning-based encoder block 1 may be designed, for example, as follows: The segmented sensor raw data SR of the camera object hypothesis OH can be converted to a latent encoding LE, for example, by using a convolutional neural network (CNN). At this time, weights are learned in the CNN. For the lidar sensor object hypothesis OH, the learning-based encoder block 1 may be designed, for example, as follows: The segmented sensor raw data SR of the lidar sensor object hypothesis OH can be converted to a latent encoding LE, for example, by using PointNet. At this time, weights are learned in PointNet.

[0037] That is, what results is that for each object hypothesis OH at each time step, we have a state and a learned latent encoding LE. The contents of this latent encoding LE are not human-interpretable. It is a maximally adequate representation of the raw sensor data SR that the model itself learned during training.

[0038] In fusion block 2, the object hypotheses OH of all sensors formed in the current time step are clustered by their respective assigned latent encoding LE to form a fused object hypothesis FOH. Then, for each time step, a graph is created. In this graph, all object hypotheses OH of a time step are nodes. Thus, each node has a state vector and a latent encoding LE of the sensor data that contains the learned appropriate representation. In the graph, all nodes are interconnected. Thus, it is a fully connected graph.

[0039] The fused object hypothesis FOH can be formed learning-based by clustering in the graph. Here, the following two variations can be used: - Calculate pairwise membership criteria between nodes in the graph. This membership criterion is learned. As in the learning-based encoder block 1, the error criterion required for this is identified only after the actual trajectory prediction and then back-propagated to the identification of the membership criterion. Based on the graph structure, e.g. graph neural networks can be used for "link prediction" and / or "edge classification". This gives pairwise probabilities that nodes belong to the same object. Based on the membership criterion, clusters of the individual nodes can be formed using standard clustering algorithms. Alternatively, it is also possible to use the learned graph clustering algorithm directly. In this alternative variation, the error criterion required for this is determined only after the actual trajectory prediction and then back-propagated to the determination of the membership criterion.

[0040] Thus, during the training process, the described learning-based clustering learns to assign object hypotheses OH to corresponding clusters such that the error of trajectory prediction is minimized. This happens when object hypotheses OH of multiple sensor modalities (e.g. camera and lidar) belonging to the same real object are also assigned to the same cluster.

[0041] In each cluster, the information of all nodes is aggregated (e.g., pooled), which corresponds to fusing multiple object hypotheses OH into one fused object hypothesis FOH. That is, the result is an aggregated latent representation of the sensor data and an aggregated state for each fused object hypothesis FOH, where the type of aggregation can be, for example, averaging.

[0042] In the tracking block 3, the fused object hypothesis FOH of the current time step and the fused object hypothesis FOH calculated in the past time steps are analyzed. In this case, it is calculated over several time steps which fused object hypothesis FOH of the past time steps belongs to which fused object hypothesis FOH of the current time step. The mutually related fused object hypothesis FOH from different time steps form a track T of the respective fused object hypothesis FOH. The track T represents the time course of the respective fused object hypothesis FOH. For tracking, a graph can be constructed which includes all fused object hypotheses FOH of the past time steps as nodes and which includes all fused object hypotheses FOH of the current time step as nodes. The feature vector of the node is again the latent encoding LE and the state. In the graph, all nodes of two successive time steps are interconnected by edges. Only for nodes connected by edges, a membership criterion is calculated. For this purpose, again a GNN (Graph Neural Network) can be used for "link prediction" and / or "edge classification". For each node of the current time step, the node of the past time step with the highest membership criterion can be identified. These nodes belong to the same object track T. As a result, a track T of fused object hypotheses FOH is obtained, i.e., the fused object hypotheses FOH are assigned to each other over multiple time steps, which may result in tracklets. Thus, at each time step, through each fused object hypothesis FOH, a track T is created in which the states of each fused object hypothesis FOH and their latent feature vectors are also assigned.

[0043] In a prediction block 4, trajectories PT of the fused object hypotheses FOH are predicted for future time steps based on their tracks T. This results in predicted trajectories PT or tracks of the various fused object hypotheses FOH.

[0044] The encoding in the encoder block 1, the clustering in the fusion block 2, and the membership formation in the tracking block 3 are performed by a learning algorithm. For training, a trajectory PT of the fused object hypothesis FOH at a future time point is predicted, and the prediction is compared with a true trajectory FT of the fused object hypothesis FOH calculated at the future time point to calculate a prediction error PE. The calculated prediction error PE is back-propagated to the encoder block 1, the fusion block 2, and the tracking block 3 for training the algorithm. Thus, the algorithms in the encoder block 1, the fusion block 2, and the tracking block 3 are optimized end-to-end at the same time.

[0045] Through Latent Encoding,LE, the trajectory prediction algorithm automatically has access to relevant sensor information that is propagated through the network.

[0046] 2 is a schematic diagram illustrating the steps of the method for predicting the trajectories of objects in the vicinity of the vehicle by backpropagation of the prediction error PE described, which, as already mentioned, is calculated by intercomparison of the predicted trajectory PT with the true trajectory FT. In the figure, the execution of this comparison is indicated by a circle.

[0047] FIG. 3 is a schematic block diagram of a system for predicting the trajectory PT of an object in the vicinity of a vehicle.

[0048] As inputs, object hypotheses OH1, OH2, OH3, OHm of different sensors are available, which may be of the same or different sensor modalities, e.g. camera, lidar, radar and / or ultrasound. In each learning-based encoder block 1, a latent encoding LE is formed for the current time step t_0 from the object hypotheses OH1 to OHm and the assigned sensor raw data SR. One and the same encoder block 1 can then be used for object hypotheses OH1 to OHm of the same sensor modality, with split weighting if necessary.

[0049] In fusion block 2, the object hypotheses OH of all sensors formed at the current time step t_0 are clustered based on their respectively assigned latent encodings LE to form a fused object hypothesis FOH.

[0050] In the tracking block 3, the fused object hypothesis FOH of the current time step t_0 and the fused object hypothesis FOH calculated in the past time steps t_(-1), t_(-T) are analyzed. The mutually related fused object hypothesis FOHs from different time steps t_0, t_(-1), t_(-T) form a track T of each fused object hypothesis FOH.

[0051] In prediction block 4, the trajectories PT of the fused object hypotheses FOH are predicted based on their tracks T for future time steps. [Prior art documents] [Patent documents]

[0052] [Patent Document 1] DE 102019215147 [Patent Document 2] DE 102019216290

Claims

1. A method for predicting a trajectory of an object in a surrounding of a vehicle, comprising: sensing sensor raw data (SR) of the surroundings of the vehicle by an environmental sensor; - the raw sensor data (SR) is pre-processed at a number of successive time steps (t_0, t_(-1), t_(-T)) to generate object hypotheses (OH); - based on said object hypotheses (OH) said raw sensor data (SR) are segmented and assigned to the respective object hypotheses (OH), - the raw sensor data (SR) belonging to each of the object hypotheses (OH) are transformed by a learning-based encoder block (1) into latent encodings (LE) and assigned as features to each of the object hypotheses (OH); - from the individual object hypotheses (OH) and the assigned features, in a fusion block (2), a fused object hypothesis (FOH) is generated by learning-based clustering; - a track (T) of each of the fused object hypotheses (FOH) is formed in a tracking block (3) by creating a learning-based assignment between the fused object hypotheses (FOH) calculated at a current time step (t_0) and the fused object hypotheses (FOH) calculated at a number of previous time steps (t_(-1), t_(-T)), Based on the track (T) of each of the fused object hypotheses (FOH), a trajectory (PT) is predicted. A method comprising:

2. The sensor raw data (SR) is sensed by multiple environmental sensors of multiple sensor modalities, and pre-processed separately for each of the sensors, and the latent encodings (LE) are identified separately for each sensor.

2. The method according to claim 1 .

3. A trajectory (PT) of the fused object hypothesis (FOH) predicted for a future time point is compared with a true trajectory (T) of the fused object hypothesis (FOH) calculated for the future time point to calculate a prediction error (PE), which is back-propagated to the encoder block (1), the fusion block (2) and the tracking block (3) for training purposes.

3. The method according to claim 1 or 2.

4. As an algorithm for predicting the trajectory (PT), a Transformer model, a recurrent neural network, or a graph neural network is used. The method according to any one of claims 1 to 3, characterized in that

5. The segmented sensor raw data (SR) of the camera object hypotheses (OH) are converted to latent encodings (LE) using a convolutional neural network, and weights are trained in the convolutional neural network. The method according to any one of claims 1 to 4, characterized in that

6. Segmented raw sensor data (SR) of the lidar sensor object hypotheses (OH) are converted to latent encodings (LE) using PointNet, and weights are trained in the PointNet. The method according to any one of claims 1 to 5, characterized in that

7. To form the Fused Object Hypothesis (FOH) in a learning-based manner, pair-wise membership criteria between nodes in the graph are calculated, and a graph neural network is used for "link prediction" and / or edge classification to generate pair-wise probabilities that nodes belong to the same object, and based on the membership criteria, clusters of the individual nodes are formed using standard clustering algorithms. The method according to any one of claims 1 to 6, characterized in that

8. A trained graph clustering algorithm is used to form the Fused Object Hypothesis (FOH) on a learning basis. The method according to any one of claims 1 to 6, characterized in that

9. For each cluster, the information of all nodes is aggregated using pooling, resulting in an aggregated latent representation of the sensor data and an aggregated state for each fused object hypothesis (FOH). The method according to any one of claims 1 to 8, characterized in that

10. For the tracking, a graph neural network is used for the "link prediction" and / or the edge classification. The method according to any one of claims 1 to 9, characterized in that

Images