A trajectory prediction system and method

Abstract

The application discloses a trajectory prediction system and method, which firstly coarsely estimates the positions of a vehicle at multiple time steps in the future according to a vehicle representation vector through a coarse trajectory generation module to form a coarse trajectory, then predicts the offset of each time step in the coarse trajectory through a trajectory optimization module containing a structure with a time induction bias, and finally calculates the trajectory according to the offset of each time step through a trajectory generation module. The system and method fuse two methods of generating a future trajectory by utilizing time correlation, effectively improve the performance of the model, make the trajectories at different time have time correlation, and further obtain better prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and particularly relates to a trajectory prediction system and method. BACKGROUND

[0002] Trajectory prediction is one of the basic tasks in the field of automatic driving, and its purpose is to predict the future positions of other intelligent agents around the ego vehicle in order to generate a safe and effective driving plan in the downstream module. At present, a single trajectory prediction method based on deep learning is mostly used. The deep learning-based method, such as the SOTA method, follows the encoder-decoder paradigm, that is, the encoder takes the historical states of all vehicles, such as position, speed, heading, and the lane, road line, traffic light and other information in the high-definition map as input, and outputs a feature vector for each vehicle. Then these vectors are taken as input to predict the multi-step future positions of each vehicle through the decoder.

[0003] The encoder can adopt a CNN network, for example, LaneGCN is a graph convolutional network (GCN) layer that encodes the scene through four different graph convolutional network (GCN) layers. In other studies, there are also some other encoding methods, for example, PGP adopts a method of traversing the lane graph to merge the lane topology, and LaPred applies an attention mechanism to the lane. HiVT designs a hierarchical way to effectively aggregate local and global information. Scene Transformer encodes spatial and temporal information in a decomposed manner, and HDGT models the driving scene as a heterogeneous graph. TPCN and its extension DCMS encode the scene in the form of point cloud, and apply MLP on the pooled instance vectors to generate future trajectories. VectorNet encodes agents and lanes as vectors on a global graph, and uses MLP to decode future trajectories according to the corresponding agent vectors. Its extensions TNT and DenseTNT generate trajectories in a two-stage manner: the first stage aims to predict the agent's goal / target, and the second stage aims to complete the entire trajectory with MLP conditioned on the end point. GOHOME and its extension THOMAS output a target heat map, and then sample a trajectory according to the selected target. LaneRCNN also follows a goal-based prediction framework, but they use a polynomial prior to complete the trajectory, and use MLP to refine the polyline. In PLOP, the mean of the distribution of future coordinates is generated in a time 4 polynomial. In the recent study Multipath++, it is shown that polynomial representation and control signal representation will lead to performance degradation compared with direct regression of coordinates.

[0004] As shown above, in most of the existing trajectory prediction methods, the decoder usually includes a multi-layer perceptron (MLP) that directly outputs a Tx2 tensor. Although models with MLP decoders dominate the leaderboard of multiple datasets, they ignore the correlation between different future time steps because there is no direct connection between the output neurons of the MLP. SUMMARY

[0005] To solve some or all of the problems in the prior art, in one aspect, the present application provides a trajectory prediction system, comprising:

[0006] a coarse trajectory generation module configured to coarsely estimate positions of a vehicle at a plurality of future time steps based on a vehicle representation vector, to form a coarse trajectory;

[0007] a trajectory optimization module comprising a structure with a temporal induction bias, the structure being configured to predict an offset for each time step in the coarse trajectory; and

[0008] a trajectory generation module configured to calculate a trajectory based on the offset for each time step.

[0009] Further, the coarse trajectory generation module comprises a multi-layer perceptron.

[0010] Further, the trajectory optimization module comprises a recurrent neural network or a 1-dimensional convolutional network.

[0011] Further, the trajectory generation module comprises an integration circuit configured to integrate the offset.

[0012] Based on the trajectory prediction system as described above, in another aspect, the present application provides a trajectory prediction method, comprising:

[0013] generating a coarse trajectory based on a vehicle representation vector by a coarse trajectory generation module;

[0014] predicting an offset for each time step in the coarse trajectory by a trajectory optimization module; and

[0015] calculating a trajectory based on the offset for each time step by a trajectory generation module.

[0016] Further, the trajectory prediction method further comprises:

[0017] calculating a difference between the trajectory output by the trajectory generation module and a real trajectory as a loss function to iteratively optimize the trajectory prediction system.

[0018] Further, the vehicle representation vector comprises:

[0019] vehicle historical state information including vehicle position, speed, and heading; and

[0020] map information, including lane information, road lines, and traffic light state information.

[0021] Further, the generation of the coarse trajectory comprises:

[0022] outputting, by a multi-layer perception, a position of the vehicle at a plurality of future time steps according to the vehicle representation vector.

[0023] Further, the trajectory optimization module comprises a recurrent neural network or a 1-dimensional convolutional network.

[0024] Further, the generation of the trajectory comprises:

[0025] integrating the offset at each time step to obtain a trajectory.

[0026] The trajectory prediction system and method provided by the present application, on the basis of the existing trajectory prediction method, fuse two methods of generating future trajectories using time correlation. Specifically, the system and method first generate a coarse trajectory using MLP, and then optimize the coarse trajectory using a structure with a time induction bias, thereby improving the performance of the model. In addition, the system also includes a trajectory generation module, which provides a target function containing time prior, accumulates velocity into displacement, still uses coordinates as the target function, so that the trajectories at different times have time correlation, thereby obtaining better prediction. BRIEF DESCRIPTION OF DRAWINGS

[0027] To further clarify the above and other advantages and features of the embodiments of the present application, a more particular description of the embodiments of the present application will be rendered by reference to specific embodiments thereof which are illustrated in the drawings. It is appreciated that these drawings depict only typical embodiments of the application and are therefore not to be considered limiting of its scope. The same or corresponding elements in the drawings are denoted by the same or similar reference signs.

[0028] Figure 1 FIG. 1 shows a structure schematic diagram of a trajectory prediction system according to an embodiment of the present application;

[0029] Figure 2 FIG. 2 shows a flow schematic diagram of a trajectory prediction method according to an embodiment of the present application; and

[0030] Figure 3 FIG. 3 shows a process schematic diagram of a trajectory prediction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, reference is made to the accompanying drawings which form a part hereof, and which are shown by way of illustration of the embodiments of the present application. However, it is understood that embodiments of the present application can be practiced without many of the specific details that are set forth in the following description. In other instances, well-known structures and operational aspects of the devices or methodologies have not been shown or described in order to avoid obscuring the application.

[0032] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0033] It should be noted that the embodiments of the present application are described in a specific order in this specification, however, this is just for the purpose of describing the specific embodiment, and does not limit the order of the steps. On the contrary, in different embodiments of the present application, the order of the steps can be adjusted according to the actual needs of adjustment.

[0034] In the present application, each module of the system according to the present application can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented by a computer program flow, for example, the module can be implemented by a code segment (such as a code segment in C, C++, etc.) stored in a storage device (such as a hard disk, a memory, etc.), wherein when the code segment is executed by a processor, the corresponding function of the module can be implemented. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, the function of the module can be implemented by hardware programming on a programmable device such as a field programmable logic gate array (FPGA), or the function of the module can be implemented by designing an application specific integrated circuit (ASIC) including a plurality of transistors, resistors and capacitors, etc. electronic devices. When the module is implemented using firmware, the function of the module can be written in the form of program code in the read-only memory such as EPROM or EEPROM of the device, and when the program code is executed by the processor, the corresponding function of the module can be implemented. In addition, some functions of the module can need to be implemented by separate hardware or cooperated with the hardware, for example, the detection function is implemented by a corresponding sensor (such as a proximity sensor, an acceleration sensor, a gyroscope, etc.), the signal transmitting function is implemented by a corresponding communication device (such as a Bluetooth device, an infrared communication device, a baseband communication device, a Wi-Fi communication device, etc.), the output function is implemented by a corresponding output device (such as a display, a speaker, etc.), and so on.

[0035] The existing trajectory prediction method using a multi-layer perceptron (MLP) can achieve good performance in simulation, but it ignores the correlation between future time steps. Specifically, from a mathematical point of view, the representation of the vehicle's feature vector H ∈ R, which includes the surrounding environment, vehicle, and its own state information, can be represented as τ ∈ R, and the predicted future trajectory τ can be represented as:

[0036] τ = W2δ(W1H).reshape(T, 2);

[0037] where δ() is a nonlinear activation function, such as the ReLU function, W1 and W2 are weight matrices, and reshape() is a conversion function that converts the output to T x 2 values to meet the requirements of the trajectory output, where T represents the length of the prediction time, and 2 represents the 2D coordinates.

[0038] Based on the expression of the future trajectory, it can be seen that each two rows of W2 contain weights for generating a position at a specific time. Since some existing trajectory prediction methods, such as SOTA, use per-step negative log-likelihood (NLL) loss or per-step mean squared error (MSE) loss, the backpropagation update of each two rows of W2 is independent. That is, this method is based on the assumption of conditional independence between future time steps, however, this assumption cannot exist in real-world scenarios, and therefore may be harmful to prediction in practical applications.

[0039] Therefore, in order to be closer to real-world scenarios, the present application also considers the correlation between future time steps during trajectory prediction. Specifically, the present application uses two methods to generate future trajectories using time correlation, first generates a rough trajectory by referring to the method in the prior art, and then optimizes the rough trajectory using a structure with a time induction bias to improve the performance of the model. During the research, the inventors found that directly fitting the velocity would reduce the performance, and adding the velocity to the displacement and still using the coordinates as the target function would result in better prediction. Based on this, in the present application, in addition to modifying the neural network structure, a target function containing a time prior is further proposed.

[0040] The scheme of the present application will be further described below in conjunction with the accompanying drawings of the embodiments.

[0041] Figure 1 A structural schematic diagram of a trajectory prediction system according to an embodiment of the present application is shown. As shown in Figure 1 A trajectory prediction system includes a rough trajectory generation module 101, a trajectory optimization module 102, and a trajectory generation module 103.

[0042] The rough trajectory generation module 101 is configured to roughly estimate the positions of the vehicle at a plurality of future time steps according to a vehicle representation vector, thereby forming a rough trajectory. The vehicle representation vector is generated by an encoder which extracts information from the historical state of the vehicle and a high-definition map, and provides a representation vector for each vehicle. Specifically, the encoder takes the historical state of the vehicle, such as position, speed, heading, and the information extracted from the map, such as lane, road line, traffic light, as input, and outputs a representation vector for each vehicle. In an embodiment of the present application, the rough trajectory generation module 101 comprises a multi-layer perception (MLP) which predicts the positions at a plurality of future time steps according to the vehicle representation vector, thereby forming a rough trajectory. The structure of the MLP and the method of generating a rough trajectory are basically the same as or similar to those in the prior art, and will not be described here.

[0043] The trajectory optimization module 102 comprises a structure with time induction bias, so that the optimization of the rough trajectory by the trajectory optimization module 102 can build the correlation between future time steps, thereby improving the performance of the system. In an embodiment of the present application, the trajectory optimization module 102 optimizes the rough trajectory by predicting the offset of each time step in the rough trajectory, wherein the offset of each time step refers to the offset of the current time step relative to the previous time step, i.e. the speed of the vehicle. In an embodiment of the present application, the trajectory optimization module 102 comprises a recurrent neural network (RNN) or a one-dimensional convolutional network (1D CNN). The recurrent neural network takes sequence data as input, recursively evolves in the sequence direction, and all nodes are connected in a chain. The one-dimensional convolutional network is a kind of feedforward neural network which comprises convolution calculation and has a deep structure, and can calculate the delay accumulation of signals. As can be seen, the use of the recurrent neural network or the one-dimensional convolutional network to predict the offset of the time step can introduce time correlation between the time steps, thereby meeting the requirements of trajectory optimization in the embodiments of the present application.

[0044] The trajectory generation module 103 is mainly configured to calculate a trajectory according to the offset of each time step. Since the output of the trajectory optimization module 102 is in the form of speed, directly fitting the speed will reduce the performance, so in an embodiment of the present application, coordinates are still used as the target function of trajectory prediction. Based on this, in an embodiment of the present application, the trajectory generation module 103 is configured to accumulate the speed into displacement, thereby obtaining a predicted trajectory. The integration process can make the trajectories at different times have time correlation. Accumulating the speed into the displacement is actually to calculate the integral of the speed output by the trajectory optimization module 102 with respect to time, so in an embodiment of the present application, the trajectory generation module 103 comprises an integration circuit.

[0045] Based on the trajectory prediction system as described above, Figure 2 and Figure 3 Figure 1 shows a flow chart and process schematic diagram of a trajectory prediction method according to an embodiment of the present application. As shown in the figure, a trajectory prediction method comprises:

[0046] First, in step 201, a rough trajectory is generated. A rough trajectory is generated based on a vehicle representation vector by a rough trajectory generation module. In an embodiment of the present application, a multi-layer perception (MLP) is used to predict a multi-step future position according to the vehicle representation vector to form a rough trajectory, wherein the vehicle representation vector is generated by an encoder according to the historical state of the vehicle such as position, speed, heading, etc., and the information extracted from the map such as lane, road line, traffic light, etc., to provide a representation vector for each vehicle. The structure of the multi-layer perception (MLP) and the method of generating a rough trajectory by the multi-layer perception (MLP) are basically the same or similar to those in the prior art, and will not be described here.

[0047] Next, in step 202, trajectory optimization. The offset of each time step in the rough trajectory is predicted by a trajectory optimization module. In an embodiment of the present application, a recurrent neural network or a 1-dimensional convolutional network is used to predict the offset of each time step in the rough trajectory, wherein the offset of each time step refers to the offset of the current time step relative to the previous time step, i.e. the speed of the vehicle; and

[0048] Finally, in step 203, a trajectory is generated. The trajectory is calculated according to the offset of each time step by a trajectory generation module. In an embodiment of the present application, the speed output by the trajectory optimization module is integrated by an integration circuit to obtain a predicted trajectory. Based on this, coordinates can be used as the objective function of trajectory prediction, and the difference between the integration result and the true trajectory can be used as the loss function to iteratively optimize the trajectory prediction system.

[0049] After testing, the prediction accuracy of the trajectory prediction method provided by the present application greatly exceeds that of existing methods on the open source Waymo Open Motion dataset.

[0050] Although the embodiments of the present application are described above, it should be understood that they are presented only as examples and not as limitations. It is obvious to those skilled in the relevant art that various combinations, variations and changes can be made without departing from the spirit and scope of the present application. Therefore, the width and scope of the present application disclosed herein should not be limited by the above disclosed exemplary embodiments, but should only be defined according to the appended claims and their equivalent replacements.

Claims

1. A trajectory prediction system, characterized in that, Comprising: a coarse trajectory generation module configured to coarsely estimate a vehicle's future positions at multiple time steps according to a vehicle representation vector, forming a coarse trajectory, wherein the vehicle representation vector is generated by an encoder that extracts information from the vehicle's history states and a high-definition map, and provides one representation vector for each vehicle; a trajectory optimization module including a structure with temporal induction bias, configured to predict an offset for each time step in the coarse trajectory; and a trajectory generation module configured to accumulate the offset for each time step into a displacement, resulting in a predicted trajectory.

2. The trajectory prediction system of claim 1, wherein, The coarse trajectory generation module includes a multi-layer perceptron.

3. The trajectory prediction system of claim 1, wherein, The trajectory optimization module includes a recurrent neural network or a 1-dimensional convolutional network.

4. The trajectory prediction system of claim 1, wherein, The trajectory generation module includes an integration circuit configured to integrate the offset.

5. A trajectory prediction method based on the trajectory prediction system according to any one of claims 1 to 4, characterized in that, Comprising steps of: generating a coarse trajectory based on a vehicle representation vector by a coarse trajectory generation module; predicting an offset for each time step in the coarse trajectory by a trajectory optimization module; and calculating a trajectory according to the offset for each time step by a trajectory generation module. Further comprising steps of:

6. The trajectory prediction method of claim 5, wherein, calculating a difference between the trajectory output by the trajectory generation module and a real trajectory, and using it as a loss function to iteratively optimize the trajectory prediction system. The vehicle representation vector includes:

7. The trajectory prediction method of claim 5, wherein, vehicle history state information including vehicle position, speed, and heading; and map information including lane information, road lines, and traffic light state information. The coarse trajectory generation includes steps of:

8. The trajectory prediction method of claim 5, wherein, outputting the vehicle's future positions at multiple time steps according to the vehicle representation vector by a multi-layer perceptron. The trajectory optimization module includes a recurrent neural network or a 1-dimensional convolutional network.

9. The trajectory prediction method of claim 5, wherein, The trajectory generation includes steps of:

10. The trajectory prediction method of claim 5, wherein, integrating the offset for each time step to obtain a trajectory. ​

Images