Method for predicting trajectory of object
Through learning-based encoder and fusion module, multiple sensor data are converted into potential encodings, forming the fusion object assumption, solving the problem of limitations of a single sensor mode, achieving more accurate trajectory prediction and autonomous vehicle behavior planning, and improving Driving safety and comfort.
Patent Information
- Application Number
- CN202380031894.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-08
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-15
AI Technical Summary
Existing methods for predicting trajectory of autonomous vehicles are usually limited to a single sensor mode, and cannot effectively integrate information from multiple sensor modes, resulting in large prediction errors and the inability to fully utilize specific information of objects around the vehicle, such as color and shape, affecting the autonomous driving system. accuracy and safety.
Using learning-based encoder modules and fusion modules, the raw data of multiple sensors is converted into potential encodings, and the fusion object assumption is formed through the graph neural network to conduct end-to-end trajectory prediction, and fuse information of multiple sensor modes, including cameras, Radar, lidar and ultrasonic sensors ensure that relevant sensor information is used for prediction.
It improves the accuracy of trajectory prediction and the reliability of autonomous vehicle behavior planning, enhances driving comfort and safety, while saving training time and supports the expansion and use of different sensor modes.
Smart Images

Figure CN118974588A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for predicting the trajectory of objects around a vehicle. Background Art
[0002] Predicting vehicles participating in traffic is an essential component of autonomous vehicles. Reliable and safe motion planning is only possible based on high-quality predictions.
[0003] DE 10 2019 215 147 A1 discloses a method for guiding an autonomous vehicle using the following steps, comprising:
[0004] -Detect the external environment of the autonomous vehicle and publish environmental information from the detected environment;
[0005] - Perform neural network-based predictions of the trajectories of traffic participants around the autonomous vehicle based on the published environmental information;
[0006] - Make rule-based predictions of the trajectories of traffic participants around the autonomous vehicle based on the published environmental information;
[0007] - Determine the risk of collision between the autonomous vehicle and surrounding traffic participants based on the neural network-based and rule-based predicted trajectories respectively;
[0008] - Based on the determined risk of collision, a trajectory predicted by a neural network or rules is selected for the respective traffic participant;
[0009] - Provides a trajectory for guiding the autonomous vehicle based on the predicted trajectories of the selected traffic participants.
[0010] Furthermore, DE 10 2019 216 290 A1 discloses a method for tracking an object based on sensor data collected by an environmental sensor for controlling a vehicle. The vehicle is equipped with an environmental sensor, and the sensor data represents the vehicle's surroundings detected by the environmental sensor. The method comprises the following steps:
[0011] -Select a subset of sensor data based on the current state of the tracked object;
[0012] - Apply machine learning methods to a selected subset of sensor data to derive information about the tracked object from the data;
[0013] -Update the current state of the tracked object based on the acquired information. Summary of the invention
[0014] The object of the present invention is to provide a new method for predicting the trajectory of objects around a vehicle.
[0015] According to the invention, this object is achieved by a method having the features of claim 1 .
[0016] Advantageous embodiments of the invention are the subject matter of the dependent claims.
[0017] A method for predicting the trajectory of objects around a vehicle according to the present invention, wherein sensor raw data from around the vehicle is collected by an environmental sensor, wherein the sensor raw data is preprocessed at multiple consecutive time steps to create object hypotheses for objects around the vehicle, wherein the sensor raw data is segmented based on the object hypothesis and assigned to the corresponding object hypothesis; wherein the sensor raw data belonging to each object hypothesis is converted into a latent code by means of a learning-based encoder module, and the latent code is assigned to the corresponding object hypothesis as a feature; wherein in a fusion module, a fused object hypothesis is created from a single object hypothesis and the assigned features through learning-based clustering; wherein in a tracking module, a trace of each fused object hypothesis is formed by creating an assignment between a fused object hypothesis determined in a current time step and a fused object hypothesis determined in multiple previous time steps in a learning-based manner, and the trajectory of each fused object hypothesis is predicted based on the trace.
[0018] Advantageously, raw sensor data of a plurality of environmental sensors of a plurality of sensor modes may be collected, each sensor may be pre-processed individually, and thereby a latent encoding may be determined individually for each sensor.
[0019] In one embodiment, the trajectory of the fused object hypothesis predicted for a future time point is compared with the actual trajectory of the fused object hypothesis determined at the future time point to determine a prediction error, wherein the determined prediction error is fed back to the encoder module, the fusion module, and the tracking module for training.
[0020] In one embodiment, two or more of a camera, a radar sensor, a lidar sensor, and an ultrasonic sensor are used as sensor modes.
[0021] In one embodiment, a transformer model, a recurrent neural network (RNN), or a graph neural network (GNN) is used as an algorithm to predict the trajectory.
[0022] In one embodiment, the patched sensor raw data of the camera's object hypothesis is converted into a latent code using a convolutional neural network (CNN), where weights are learned in the convolutional neural network.
[0023] In one embodiment, the patched sensor raw data of the object hypothesis of the lidar sensor is converted into a latent code by PointNet, where the weights are learned in PointNet.
[0024] In one embodiment, pairwise affiliation metrics between nodes in a graph are calculated to form a fused object hypothesis based on learning, wherein a graph neural network is used to perform "link prediction" and / or edge classification to determine the likelihood / probability that pairs of nodes belong to the same object, wherein clusters of individual nodes are formed based on the affiliation metrics using a standard clustering algorithm.
[0025] In another embodiment of the method, the learned graph clustering algorithm is used to form a fused object hypothesis based on learning.
[0026] In one embodiment, information from all nodes is aggregated into each cluster via pooling, such that each fused object hypothesis forms an aggregated latent representation of the sensor data and an aggregated state.
[0027] In one embodiment, the graph neural network used for tracking is used for "link prediction" and / or edge classification.
[0028] The learning-based trajectory prediction method has proven to be particularly accurate. The present invention is the first to introduce how to use a learned end-to-end approach to perform fusion, tracking and prediction. The learned end-to-end approach ensures that relevant sensor information for individual object hypotheses can also be used for prediction.
[0029] In known trajectory prediction methods for tracking vignettes (time series of 2D xy coordinates of a single car in the scene), the tracking vignettes come from an upstream stack that already handles perception, tracking and fusion of a single agent. The disadvantage of these approaches is that only the tracking vignettes are used as input information for the prediction. Unlike these approaches, the solution according to the invention does not lose sensor-specific information (such as the recognized vehicle color or shape). This can therefore be advantageous, since the shape is likely to be relevant for the prediction: for example, a sports car behaves differently kinematically than a classic family sedan, so the presence of this type of information can also be used to advantage in the prediction.
[0030] Other trajectory prediction approaches process only raw sensor data from a single sensor modality, where object recognition and prediction are learned in an end-to-end manner. The problem here is that these approaches are always limited to a single sensor modality. This means that objects are usually only recognized using a lidar scanner, and then tracked and predicted over a period of time. Unlike these approaches, the solution according to the present invention meets an important need for autonomous driving systems by taking into account multiple sensor modalities (camera, lidar, radar, ultrasonic). All of these sensor modes can generate valuable information, which can be used simultaneously by the solution according to the present invention.
[0031] According to the method of the present invention, the detection results of any number of independent sensor modes can be used, and these detection results can be fused, tracked over a period of time, and a prediction result can be generated. Here, the end-to-end learning-based approach ensures that relevant sensor information (which information is relevant to the prediction and how to extract this information) can also be used for prediction.
[0032] According to the method of the invention, trajectory prediction can be improved by using relevant sensor information in the form of latent codes. Here, what information is relevant is learned, rather than determined by artificially created indicators. Better prediction means that the behavior of the autonomous vehicle can be better planned. This improves driving comfort and safety. The end-to-end approach avoids the training of individual components and can be trained as a whole. This saves training time. Recognized objects of different sensor modalities can be used without problems. Furthermore, it can be extended to use any number of sensors and any sensor modalities. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Embodiments of the present invention are explained in more detail below with reference to the accompanying drawings.
[0034] in:
[0035] Figure 1 A flow chart of a method for predicting the trajectory of objects around a vehicle is shown.
[0036] Figure 2 A flow chart of a method for predicting the trajectory of objects around a vehicle is shown, and
[0037] Figure 3 A block diagram of a system for predicting trajectories of objects around a vehicle is shown.
[0038] In all the figures, parts which correspond to one another are provided with the same reference numerals. DETAILED DESCRIPTION
[0039] The present invention relates to a method for predicting the trajectory of objects around a vehicle. The vehicle has multiple sensors for detecting the surrounding environment, such as at least one camera, at least one radar sensor, at least one lidar sensor and / or at least one ultrasonic sensor. The present invention is based on the fact that the raw sensor data from the sensor has been preprocessed. This preprocessing is performed separately for each sensor (for a single sensor). Object hypotheses are created during the preprocessing. Object hypotheses refer to data sets extracted from sensor raw data and containing object information. This information includes, for example, information such as object type (pedestrian, vehicle) and object state (the position of the object in the common coordinate system of all sensors, the size of the object). Part of the object hypothesis is the state vector and the raw sensor data.
[0040] For example, an object hypothesis can be determined based on the data detected by a camera, which includes the image of the detected object and the respective position of each object in the coordinate system. An object hypothesis can also be determined based on the data detected by a radar sensor, which includes the detected reflection point, position and speed of the detected object in the coordinate system (due to the Doppler effect, radar can also measure speed).
[0041] Therefore, on the one hand, the object hypothesis has a state vector (hereinafter referred to as "state", called "State" in English), which contains information about the object hypothesis. The state reflects at least the position and size of the object hypothesis in a unified coordinate system (the position and size of the object for which the object hypothesis is created). Based on this, other sensor-specific sizes can also become part of the object hypothesis state. For example, radar detection can also have speed. In addition, the raw sensor data of the object is segmented according to the object hypothesis (and its size) and assigned to the corresponding object hypothesis. For example, when using a camera or lidar sensor, the pixels of the identified vehicle are extracted (semantic extraction of the pixels of the identified vehicle is performed).
[0042] This preprocessing is repeated over several consecutive time steps.
[0043] The raw sensor data belonging to each object hypothesis is converted into latent codes using a learning-based encoder and assigned to the corresponding object hypothesis as features.
[0044] Through (learning-based) clustering, a fused object hypothesis is created from individual object hypotheses and the assigned features.
[0045] In the next step, a trace of individual fused object hypotheses is formed by creating assignments (in a learning-based manner) between the fused object hypotheses determined in a current time step and the fused object hypotheses determined in multiple previous time steps.
[0046] In the next step, the trajectory of each fused object hypothesis is predicted based on the traces. The following algorithms can be considered for predicting trajectories: Transformer, Recurrent Neural Network (RNN), Graph Neural Network (GNN).
[0047] Figure 1 A flow chart of a method for predicting trajectories of objects around a vehicle.
[0048] In the encoder module 1, a latent code LE is formed according to the object hypothesis OH determined in the preprocessing and the assigned sensor raw data SR. Among them, a latent code LE is formed for each object hypothesis OH determined in the current time step and is assigned to the corresponding object hypothesis OH as a feature. The latent code LE is a value from a preset finite value set. The sensor raw data SR is data from an infinite value set. Therefore, the sensor raw data SR from the infinite value set can be mapped to a value from a finite value set by encoding. For the object hypothesis OH of the camera, for example, the learning-based encoder module 1 can be designed as follows: For example, a convolutional neural network (CNN) can be used to convert the block sensor raw data SR of the object hypothesis OH of the camera into a latent code LE. The weights will be learned in the CNN. For the object hypothesis OH of the lidar sensor, for example, the learning-based encoder module 1 can be designed as follows: For example, the block sensor raw data SR of the object hypothesis OH of the lidar sensor can be converted into a latent code LE through PointNet. The weights will be learned in PointNet.
[0049] This results in a state and a learned latent code LE for each object hypothesis OH at each time step. The content of these latent codes LE cannot be interpreted by humans. This is a representation of the raw sensor data SR that is as appropriate as possible, and is learned by the model itself during training.
[0050] In fusion module 2, the object hypotheses OH of all sensors formed in the current time step are clustered according to the potential encoding LEs assigned to them, and a fused object hypothesis FOH is formed. Here, a graph is created for each time step. In this graph, all object hypotheses OH of this time step are nodes. Therefore, each node has a state vector and a potential encoding LE, which contains the learned appropriate representation of the sensor data. In the graph, all nodes are connected to each other. Therefore, it is a fully connected graph.
[0051] Through the clustering in the graph, it is possible to form a fused object hypothesis FOH in a learning-based manner. There are two options for this:
[0052] - Pairwise attachment measures between nodes in the graph are calculated. The attachment measures are learned. As in the case of the learning-based encoder module 1, the error measure required here can only be determined after the actual trajectory prediction and then passed back to determine the attachment measures. Due to the structure of the graph, graph neural networks can be used to perform "link prediction" and / or "edge classification" etc. This gives the pairwise probabilities that the nodes belong to the same object. Based on the attachment measures, clusters of single nodes can be formed using standard clustering algorithms.
[0053] Alternatively, the learned graph clustering algorithm can also be used directly. In this alternative method, the error measurement required here can also be determined only after the actual trajectory prediction and then passed back to determine the associated metric value.
[0054] Therefore, during training, the learning-based clustering method learns how to assign object hypotheses OH to corresponding clusters in order to minimize the error in trajectory prediction. This happens when object hypotheses OH belonging to the same real object in multiple sensor modalities (such as camera and lidar) are also assigned to the same cluster.
[0055] The information of all nodes is aggregated (e.g., pooled) for each cluster. This is equivalent to fusing multiple object hypotheses OH into a fused object hypothesis FOH. This allows each fused object hypothesis to form an aggregated potential representation of sensor data and an aggregated state. This state can be imagined as an aggregated type of averaging.
[0056] In the tracking module 3, the fused object hypothesis FOH of the current time step and the fused object hypothesis FOH determined in the previous time step are analyzed. Among them, it is necessary to determine within multiple time steps which of the fused object hypothesis FOH of the current time step the fused object hypothesis FOH of the previous time steps belongs to. The fused object hypothesis FOHs from different time steps and belonging to each other constitute the trace T of each fused object hypothesis FOH. The trace T describes the time process of each fused object hypothesis FOH. In order to track, a graph can be constructed, which contains all fused object hypothesis FOHs of the previous time step as nodes and all fused object hypothesis FOHs of the current time step as nodes. The feature vector of the node is also the potential code LE and state. In the graph, all nodes of two consecutive time steps are connected to each other by edges. The attached metric value is determined only for the nodes connected to a certain edge. Similarly, a graph neural network (GNN) can be used for "link prediction" and / or "edge classification". For each node of the current time step, the node with the highest attached metric value in the previous time step can be determined. These nodes belong to the trace T of the same object. This results in a track T consisting of the fused object hypotheses FOH. This means that the fused object hypotheses FOH can be assigned to each other in multiple time steps to form tracking fragments. Accordingly, a track T is formed, and the states of each fused object hypothesis FOH and its latent feature vector are also assigned to this track T in each corresponding time step of each fused object hypothesis FOH.
[0057] In the prediction module 4, the track T of the fused object hypothesis FOH is used to predict its track PT in the future time step. In this way, the predicted track PT or track of different fused object hypotheses FOH is obtained.
[0058] The encoding in encoder module 1, the clustering in fusion module 2, and the establishment of affiliation in tracking module 3 are all performed by learning algorithms. During the training process, the trajectory PT of the fused object hypothesis FOH at a certain point in the future is predicted, and the prediction result is compared with the true trajectory FT of the fused object hypothesis FOH determined at a certain point in the future to determine the prediction error PE. The determined prediction error PE is fed back to encoder module 1, fusion module 2, and tracking module 3 to train the algorithm. As a result, the algorithms in encoder module 1, fusion module 2, and tracking module 3 are optimized end-to-end at the same time.
[0059] Through latent encoding LE, the trajectory prediction algorithm can automatically obtain relevant sensor information transmitted through the network.
[0060] Figure 2The process of the method for predicting the trajectory of the object around the vehicle by using the feedback of the prediction error PE is schematically shown. As mentioned above, the prediction error PE is determined by comparing the predicted trajectory PT and the true trajectory FT. The execution of this comparison is indicated by a circle in the figure.
[0061] Figure 3 A block diagram of a system for predicting a trajectory PT of an object around a vehicle is shown.
[0062] The object hypotheses OH1, OH2, OH3, OHm of different sensors are used as input values, which can come from the same or different sensor modes, such as camera, lidar, radar and / or ultrasound. In each learning-based encoder module 1, a latent code LE is formed based on the object hypotheses OH1 to OHm and the sensor raw data SR assigned to the current time step t_0. The same encoder module 1 can be used for the object hypotheses OH1 to OHm of the same sensor mode, and different weights can be used if necessary.
[0063] In the fusion module 2, the object hypotheses OH of all sensors formed in the current time step t_0 are clustered according to their assigned potential encoding LEs, and a fused object hypothesis FOH is formed.
[0064] In the tracking module 3, the fused object hypothesis FOH of the current time step t_0 and the fused object hypothesis FOH determined in the previous time steps t_(-1) and t_(-T) are analyzed. The fused object hypotheses FOH from different time steps t_0, t_(-1), t_(-T) and belonging to each other constitute the trace T of each fused object hypothesis FOH.
[0065] In the prediction module 4, the trace T of the fused object hypothesis FOH is used to predict its trajectory PT in future time steps.
Claims
1. A method for predicting the trajectory of objects around a vehicle, wherein: Collecting raw sensor data (SR) from the vehicle’s surroundings through environmental sensors, It is characterized in that - pre-processing of said sensor raw data (SR) in a plurality of consecutive time steps (t_0, t_(-1), t_(-T)) in order to create an object hypothesis (OH), - splitting the sensor raw data (SR) according to the object hypothesis (OH) and assigning them to the corresponding object hypothesis (OH), - using a learning-based encoder module (1), converting said sensor raw data (SR) belonging to each object hypothesis (OH) into latent codes (LE) and assigning them as features to the corresponding object hypothesis (OH), - In the fusion module (2), a fused object hypothesis (FOH) is created from the individual object hypotheses (OH) and the assigned features through learning-based clustering, - in the tracking module (3), forming a track (T) of the respective fused object hypotheses (FOH) by creating an assignment in a learning-based manner between the fused object hypothesis (FOH) determined in a current time step (t_0) and the fused object hypotheses (FOH) determined in a plurality of previous time steps (t_(-1), t_(-T)), - predicting the trajectory (PT) of each fused object hypothesis (FOH) from the trace (T).
2. The method according to claim 1, It is characterized in that Sensor raw data (SR) of multiple environmental sensors of multiple sensor modes are collected, each sensor is pre-processed individually, and thereby a latent code (LE) is determined individually for each sensor.
3. The method according to claim 1 or 2, It is characterized in that The trajectory (PT) of the fused object hypothesis (FOH) predicted for a future time point is compared with the true trajectory (PT) of the fused object hypothesis (FOH) determined at the future time point to determine a prediction error (PE), wherein the determined prediction error (PE) is fed back to the encoder module (1), the fusion module (2) and the tracking module (3) for training.
4. The method according to any one of the preceding claims, It is characterized in that The trajectory (PT) is predicted using a transformer model, a recurrent neural network, or a graph neural network as an algorithm.
5. The method according to any one of the preceding claims, It is characterized in that The patched sensor raw data (SR) of the camera's object hypothesis (OH) is converted into a latent code (LE) using a convolutional neural network, where the weights are learned in the convolutional neural network.
6. The method according to any one of the preceding claims, It is characterized in that The chunked sensor raw data (SR) of the object hypothesis (OH) of the LiDAR sensor is converted into latent encodings (LE) through PointNet, where the weights are learned in PointNet.
7. The method according to any one of the preceding claims, It is characterized in that Pairwise affiliation metrics between nodes in the graph are computed to form a fused object hypothesis (FOH) based on learning, where a graph neural network is used to perform "link prediction" and / or edge classification so that the probability of nodes belonging to the same object appears in pairs, where clusters of individual nodes are formed based on the affiliation metrics using a standard clustering algorithm.
8. The method according to any one of claims 1 to 6, It is characterized in that The learned graph clustering algorithm is used to form a fused object hypothesis (FOH) based on learning.
9. The method according to any one of the preceding claims, It is characterized in that The information of all nodes is aggregated into each cluster through pooling, so that each fused object hypothesis (FOH) forms an aggregated latent representation of the sensor data and an aggregated state.
10. The method according to any one of the preceding claims, It is characterized in that Graph neural networks for tracking are used for "link prediction" and / or edge classification.
