A vehicle trajectory prediction method based on heterogeneous node space-time domain perception
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2023-03-15
- Publication Date
- 2026-05-26
Smart Images
Figure CN116176627B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a vehicle trajectory prediction method based on heterogeneous node spatiotemporal perception, and belongs to the field of deep learning autonomous driving technology. Background Technology
[0002] Trajectory prediction is crucial for the safe operation of autonomous vehicles in dynamic environments. An ideal autonomous driving algorithm should be able to predict the future states of nearby road users, especially surrounding vehicles, possessing predictive driving capabilities similar to human drivers. Predicting the future positions of vehicles and pedestrians is a stochastic, multidimensional problem influenced by numerous factors. This complexity exceeds the capabilities of mathematical models; therefore, existing excellent methods tend to use data-driven approaches to achieve prediction, such as deep learning and reinforcement learning.
[0003] Common trajectory prediction methods based on deep learning primarily study the correlation between a target's past and future trajectories. They can be categorized into map-free and map-based trajectory prediction methods, depending on whether the static surrounding environment is considered. Map-free trajectory prediction typically considers each target's past trajectory and the current trajectories of other similar targets within a certain range, using these factors as input to a deep neural network, hoping the network will understand the correlation between past and future trajectories. However, this method ignores many factors influencing the trajectory and lacks prior knowledge constraints on the neural network's output, leading to inaccurate predictions.
[0004] Map-based trajectory prediction methods typically utilize high-precision map information to obtain lane line information and drivable area information in the local area surrounding the target vehicle as additional prediction factors. More comprehensive methods also consider traffic light and pedestrian location information. The former can feed this map information as an image type into the network, leveraging the advantages of neural networks in automatic feature integration and strong model representation capabilities in image vision. Combined with the recurrent neural network's ability to understand the target trajectory, it jointly infers the target's future trajectory. The latter treats traffic lights, pedestrians, lane lines, and other information as a series of heterogeneous graph nodes, forming a heterogeneous graph together with the vehicle's trajectory location. It uses heterogeneous graph neural networks (HGNNs) to understand the sparsity and multi-level characteristics of heterogeneous nodes, effectively mining and utilizing multiple types of information and integrating them into a specific prediction problem, effectively inferring the target's future trajectory location. However, these methods all use high-precision maps that are pre-collected and stored offline or intermittently updated periodically. Due to the accuracy and richness of their information, their update frequency cannot adequately meet the needs of autonomous driving tasks in high-dynamic environments, and acquiring high-precision maps over large areas is costly. In practical applications of prediction algorithms, autonomous vehicles often lose some accuracy in their predictions for environments outside the coverage area of high-precision maps or for environments where high-precision maps are updated slowly and cannot accurately represent the real road conditions. Summary of the Invention
[0005] To overcome the shortcomings of existing research, this invention provides a vehicle trajectory prediction method based on heterogeneous node spatiotemporal domain perception. It builds an online perception map generated from the perspective perception results of the driving vehicle and uses a spatiotemporal domain attention mechanism to complete the vehicle trajectory prediction task by understanding and fusing different types of information. It can effectively obtain map information within the local range of the target vehicle without using high-precision maps, and complete the trajectory prediction while ensuring high robustness and estimation accuracy, which is more in line with the needs of practical applications.
[0006] The specific steps of a vehicle trajectory prediction method based on heterogeneous node spatiotemporal domain perception are as follows:
[0007] Step 1: Process the 3D perception results from sensor data to reduce the dimensionality to the BEV perspective to generate an online perception map and extract obstacle location information.
[0008] Step 2: Construct a heterogeneous node perception network, comprising a parallel dynamic attention network and a static attention network. The dynamic attention network includes a spatial self-attention module and dynamically allocated fully connected layers, while the static attention network comprises an 18-layer residual neural network with three fully connected layers.
[0009] Step 3: Construct a spatiotemporal trajectory sequence understanding network, which includes a vehicle spatiotemporal attention module, a pedestrian group social attention module, and a feature fusion trajectory generation network. The attention modules are all attention mechanisms that generate trajectory sequences of the same dimension from the input trajectory sequence. The difference is that the temporal attention inputs the past trajectory information of a single obstacle, while the social attention selectively inputs the current trajectory of other obstacles in the social area of a single obstacle. The feature fusion trajectory generation network includes a VAE network and three fully connected layers.
[0010] Step 4: Set the learning rate and training period. Randomly extract a single scene from the dataset and input it sequentially into each keyframe. Record the online perception map and location records processed in Step 1 into the perception network in Step 2 to generate perception feature data. Input the perception features into the understanding network in Step 3 to generate the final vehicle trajectory prediction result. Compare the prediction result with the location information records from Step 1 to verify the network effect.
[0011] Preferably, the method for generating online perception maps through 3D perception dimensionality reduction specifically uses the LiDAR of an autonomous vehicle as the basis for local range perception, extracts radar scan points within ±80m, and combines sensor-fused 3D target detection and semantic segmentation perception data to generate an online map with target annotations and semantic mapping. When predicting a target vehicle, the coordinate space is transformed based on the target position reported by the target detection framework, and the online map within ±30m of the target is extracted as continuous feature input. The map generation framework simultaneously distinguishes between individual vehicle positions as a trajectory sequence over continuous time and does not distinguish between individual pedestrian positions as pedestrian group distribution information; these are uniformly used as discrete feature input.
[0012] Preferably, the spatiotemporal domain trajectory sequence understanding network has the following specific modules:
[0013] (a) The spatiotemporal attention module extends the trajectory data to a two-dimensional index space for all vehicles perceived within a single keyframe, based on individual differences and time variations. Then, it encodes the trajectory according to time and individual differences, embedding it into an 8-layer transformer structure. The specific formula and method are as follows:
[0014]
[0015]
[0016]
[0017] head i =Attention(Q) i ,K i V i )
[0018] MultiHead(Q,K,V)=Concact(head1,head2,…head s W o
[0019] Where j is the number of the vehicle to be predicted, i is the number of the multi-head attention, n is the number of target vehicles, s is the number of multi-head attention, and t is the maximum number of past trajectories counted.
[0020] (b) Pedestrian group social attention module: For pedestrian location data that does not distinguish individuals and does not record historical locations, the group affiliation is determined based on the distance between each pedestrian's location, and the number of groups is determined based on the total number of pedestrians. Thus, the groups are encoded and embedded in a 2-layer transformer structure.
[0021] Preferably, the dataset used in step one is the Nuscenes dataset. The provided high-precision map and metadata of the visual sensor are not used in the prediction process. The data sources of this method are mainly the results of object detection and semantic segmentation, as well as the recorded target location information.
[0022] As a preferred option, the learning rate of the network model in step four is 0.0003, because the training individual is a target vehicle in a keyframe of a scene, so only one keyframe must be trained at a time, and the training cycle is 150.
[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0024] This invention proposes a novel map information processing framework and utilizes the perceptual information provided by this framework to design an attention-based network framework. This framework analyzes heterogeneous obstacle categories in the spatiotemporal domain and outputs the predicted future trajectory of the vehicle. Based on the proposed map information processing framework, autonomous vehicles can acquire a certain level of online map perception capability while avoiding the use of high-precision maps, thus enabling a wider range of applications. Based on the proposed network framework, feature information from online perceived maps can be effectively extracted, and the motion states of pedestrians and vehicles can be analyzed discriminatively. By integrating multiple traffic factors to analyze vehicle trajectories, better predictive performance can be achieved. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a schematic diagram of the online sensing map generation and data distribution process of the vehicle trajectory prediction method based on heterogeneous node spatiotemporal domain perception according to the present invention.
[0027] Figure 2 This is a schematic diagram of the spatiotemporal domain trajectory sequence understanding module of the present invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] A vehicle trajectory prediction method based on heterogeneous node spatiotemporal domain perception specifically includes the following steps:
[0030] Step 1: To effectively obtain the information needed to generate online perception maps, this project can use the Nuscenes dataset. This dataset collected driving data for 1000 scenes in Boston and Singapore, USA. Each scene is a short video of about 20 seconds, using 6 visual sensors, one LiDAR, and one millimeter-wave radar as inputs. It provides real data for common vision tasks in autonomous driving, such as 3D object detection, image and point cloud semantic segmentation, and trajectory prediction. Figure 1 This framework first reads perception data such as 3D object detection, semantic segmentation, and vehicle location from the dataset. Using the LiDAR of the autonomous vehicle as the basis for localized perception, it extracts radar scan points within ±80m and combines them with sensor-fused 3D object detection and semantic segmentation perception data to generate an online map with target annotations and semantic mapping. When predicting a target vehicle, it performs coordinate space transformation based on the target location reported by the object detection framework, extracting the online map within ±30m of the target as continuous feature input. The map generation framework simultaneously records individual vehicle locations as continuous time trajectory sequences and pedestrian locations without distinguishing between individuals as pedestrian group distribution information; these are uniformly used as discrete feature input.
[0031] Step 2: Before feeding the image and target past trajectory to the network framework, the map and trajectory are simultaneously rotated randomly to increase data diversity and enhance network robustness. Then, a dynamic attention network directly uses the large-size complete map perceived by the vehicle to encode and embed the attention module to obtain feature extraction results across the entire domain. Simultaneously, a static attention network processes a limited number of online perception maps based only on the number of target vehicles perceived by the vehicle in a single keyframe, and uses an 18-layer residual network to input in parallel to obtain feature extraction results for a local range.
[0032] Step 3, as follows Figure 2 The spatiotemporal trajectory sequence understanding network has the following specific modules:
[0033] (a) The spatiotemporal attention module extends the trajectory data to a two-dimensional index space for all vehicles perceived within a single keyframe, based on individual differences and time variations. Then, it encodes the trajectory according to time and individual differences, embedding it into an 8-layer transformer structure. The specific formula and method are as follows:
[0034]
[0035]
[0036]
[0037] head i =Attention(Q) i ,K i V i )
[0038] MultiHead(Q,K,V)=Concact(head1,head2,…head s W o
[0039] Where j is the predicted vehicle number, i is the multi-head attention number, n is the number of target vehicles, s is the number of multi-head attention points, and t is the maximum number of past trajectories counted.
[0040] (b) Pedestrian group social attention module: For pedestrian location data that does not distinguish individuals and does not record historical locations, the group affiliation is determined based on the distance between each pedestrian location, and the number of groups is determined based on the total number of pedestrians. Thus, the groups are encoded and embedded in a 2-layer transformer structure.
[0041] Step 4: Set the learning rate and training period. Randomly extract a single scene from the dataset and input it sequentially into each keyframe. Record the online perception map and location records processed in Step 1 into the perception network in Step 2 to generate perception feature data. Input the perception features into the understanding network in Step 3 to generate the final vehicle trajectory prediction result. Compare the prediction result with the location information records from Step 1 to verify the network effect.
[0042] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.
Claims
1. A vehicle trajectory prediction method based on heterogeneous node spatiotemporal domain perception, characterized in that: Includes the following steps: Step 1: Process the 3D perception results from sensor data acquisition to reduce the dimensionality to the BEV perspective to generate an online perception map and extract obstacle location information. In Step 1, the LiDAR of the autonomous vehicle is used as the basis for local range perception. The radar scan points within ±80m are extracted and combined with sensor fusion of 3D target detection and semantic segmentation perception data to generate an online map with target annotations and semantic mapping. When predicting a target vehicle, the coordinate space is transformed according to the target position reported by the target detection, and the online map within ±30m around the target is extracted as continuous feature input. The map generation framework will simultaneously distinguish between recording individual vehicle positions as a trajectory sequence in continuous time and recording pedestrian positions without distinguishing between individuals as pedestrian group distribution information, and uniformly use them as discrete feature input. Step 2: Construct a heterogeneous node perception network, comprising a parallel dynamic attention network and a static attention network. The dynamic attention network in Step 2 includes a spatial self-attention module and dynamically allocated fully connected layers. The static attention network includes an 18-layer residual neural network with three fully connected layers. In Step 2, the dynamic attention network directly uses the large-size complete map encoding perceived by the vehicle to embed into the attention module to obtain feature extraction results. Simultaneously, the static attention network processes a limited number of online perception maps based only on the number of target vehicles perceived by the vehicle in a single keyframe, and uses the 18-layer residual network as parallel input to obtain feature extraction results. Step 3: Construct a spatiotemporal trajectory sequence understanding network, including a vehicle spatiotemporal attention module, a pedestrian group social attention module, and a feature fusion trajectory generation network; the specific modules of the spatiotemporal trajectory sequence understanding network in Step 3 are as follows: The vehicle spatiotemporal attention module extends the trajectory data to a two-dimensional index space for all vehicles perceived within a single keyframe, based on individual differences and time differences. Then, it encodes the trajectory based on time information and individual difference information and embeds it into an 8-layer transformer structure. The pedestrian group social attention module, for pedestrian location data that does not distinguish individuals or record historical locations, determines the group affiliation based on the distance between each pedestrian's location, determines the number of groups based on the total number of pedestrians, encodes the groups and embeds them into a 2-layer transformer structure; Step 4: Set the learning rate and training period, randomly extract a single scene from the dataset, and input it sequentially into each keyframe. The online perception map and location records processed in Step 1 are then fed into the perception network in Step 2 to generate perception feature data. The perception features are input into the understanding network in step three to generate the final vehicle trajectory prediction result; the prediction result is compared with the location information recorded in step one to verify the network effect.
2. The vehicle trajectory prediction method based on heterogeneous node spatiotemporal perception according to claim 1, characterized in that: The feature fusion trajectory generation network in step three includes a VAE network and three fully connected layers.