Target-driven multimodal trajectory prediction method, vehicle, equipment and medium

Abstract

This invention discloses a target-driven multimodal trajectory prediction method, vehicle, device, and medium. The method includes: acquiring a high-precision autonomous driving dataset, including trajectory data of road participants and a vectorized high-precision map; obtaining trajectory features and map features respectively; fusing features using LaneGCN and a spatial attention layer; and predicting K for each road participant. e Each candidate target is identified and a confidence score is calculated. The candidate target with the highest confidence score is selected as the predicted target, and the area within a certain range of the predicted target is considered the GOI region. The GOI region is encoded to obtain the final features. The final features are then used to predict K values ​​for each road participant. a The system generates several predicted trajectories and their corresponding confidence scores. After training, the predicted trajectory with the highest confidence score is selected as the final trajectory prediction result. This invention can better handle complex and uncertain traffic conditions and improve the accuracy of trajectory prediction.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving technology, specifically relating to a target-driven multimodal trajectory prediction method, vehicle, equipment, and medium. Background Technology

[0002] Autonomous driving technology is an innovative technology that uses computer and sensor technologies to enable cars to drive independently. The goal of this technology is to enable vehicles to operate safely without human driver intervention, thereby improving traffic efficiency, reducing traffic accidents, and providing more convenient modes of transportation. Trajectory prediction plays a crucial role in autonomous driving; it is an important component of autonomous driving systems and contributes to achieving safe, efficient, and intelligent autonomous driving.

[0003] Trajectory prediction technology is primarily used to predict a vehicle's future trajectory, enabling the vehicle to make corresponding decisions and control its driving. In autonomous driving systems, trajectory prediction is not only crucial for vehicle safety and stability but also closely related to functions such as path planning and obstacle avoidance. Only through accurate trajectory prediction can vehicles better avoid traffic obstacles, predict the driving intentions of other vehicles, and maintain safe driving distances and speeds. Therefore, trajectory prediction technology is of great significance to the performance and reliability of autonomous driving systems. This technology has wide applications in various fields, including autonomous driving, drone navigation, logistics, and aerospace.

[0004] Previous trajectory prediction technologies focused solely on historical vehicle trajectories and driving environments, primarily concentrating on interactions between vehicles and between vehicles and their environment within the observation window. However, they rarely explored vehicle driving behavior within the prediction window, such as whether a vehicle would evade a collision due to adherence to driving ethics at a future point in time. Furthermore, human intentions are highly complex in real-world driving environments. For instance, at an intersection, a driver may have multiple choices, such as stopping, accelerating, or turning; different actions can lead to completely opposite outcomes. This makes understanding behavioral intentions crucial for future trajectory prediction.

[0005] With the development of autonomous driving and related fields, the requirements for trajectory prediction technology are becoming increasingly stringent, especially when dealing with complex, diverse, and uncertain traffic scenarios. Goal-driven multimodal trajectory prediction technology has become at the forefront of this field and plays a crucial role in meeting these high demands.

[0006] Target-driven multimodal trajectory prediction is an advanced trajectory prediction method designed to more accurately predict the future trajectories of moving targets (such as vehicles, pedestrians, and bicycles) to improve decision-making and safety in autonomous driving and related fields. The core idea of ​​this technology is to combine multimodal data (such as sensor data) with the target's intent and possible behaviors to generate a variety of possible trajectory predictions. These trajectory predictions can cover a wide range of possibilities under different conditions, helping the system better understand its surroundings and make smarter decisions. Multimodal trajectory prediction technology has multiple advantages, including data fusion, diversity, uncertainty handling, target intent understanding, real-time performance, interactivity considerations, and a wide range of applications, making it one of the key technologies in autonomous driving and related fields. This technology helps improve the intelligence and safety of transportation systems, thus providing more possibilities for future mobility. Summary of the Invention

[0007] The purpose of this invention is to provide a target-driven multimodal trajectory prediction method, vehicle, device, and medium that can more accurately predict the behavior of other vehicles, pedestrians, and bicycles, enabling autonomous driving systems to better avoid potential dangerous situations, thereby improving road safety and reducing traffic accidents.

[0008] The first aspect of the present invention provides a target-driven multimodal trajectory prediction method, the method comprising:

[0009] Acquire a high-precision autonomous driving dataset, which includes trajectory data of road participants and vectorized high-precision maps;

[0010] Historical trajectory data of road participants using U-Net Perform feature extraction operations to obtain trajectory features ;

[0011] A lane graph is constructed based on a vectorized high-precision map, and convolution operations are performed on the lane graph using LaneGCN to obtain map features. ;

[0012] Fusing trajectory features using LaneGCN and spatial attention layer and map features To capture all interaction information between road participants and roads, and obtain fused features. ;

[0013] Based on the fused features Predictions for each road participant Candidate targets And apply MLP to each candidate target Calculate the confidence score and use The function calculates the error between the best candidate target and the true target.

[0014] The candidate target with the highest confidence score is selected as the prediction target, and the region within a certain range of the prediction target is regarded as the GOI region. The GOI region is encoded to obtain the final feature X.

[0015] Using the final feature X, predict the outcome for each road participant. The predicted trajectories and their corresponding confidence scores are calculated; the classification error is calculated by applying the max-margin loss function. The function calculates the displacement error between all predicted trajectories and their corresponding actual ground trajectory coordinates at each time step.

[0016] After training, the predicted trajectory with the highest confidence score is selected as the final trajectory prediction result.

[0017] The above scheme further includes the following method:

[0018] The data is preprocessed to clean and process the raw data, handle missing values, outliers and noise, and perform coordinate system transformation and unit standardization.

[0019] In the above scheme, historical trajectory data of road participants are analyzed using U-Net. Perform feature extraction operations to obtain trajectory features ,include:

[0020] The trajectory data of road participants is divided into observed historical trajectory data. and the actual trajectory data that needs to be predicted Using U-Net encoders to process historical trajectory data Feature extraction is performed, and then the U-Net decoder is used to map the features extracted by the encoder back to the original data form to obtain trajectory features. .

[0021] In the above scheme, all interaction information between road participants and roads includes: road participant to road A2L, road to road L2L, road to road participant L2A, and road participant to road participant A2A.

[0022] In the above scheme, based on the fused features Predictions for each road participant Candidate targets And apply MLP to each candidate target Calculate the confidence score and use The function calculates the error between the best candidate target and the true target, including:

[0023] Applying Transformer Generate a set of candidate targets for each road participant:

[0024]

[0025] in, G For the candidate target set, For the number of road participants, Indicates the first The first road participant's One candidate target;

[0026] To make candidate targets closer to the real target, a method based on MLP is designed. M The classification branch formed by (·) is used to output the confidence score corresponding to each candidate target:

[0027]

[0028] in, For the set of confidence scores, Indicates the first The first road participant's Confidence scores for each candidate target;

[0029] Calculate the difference between candidate targets and the real target This is used to further improve the accuracy of target prediction.

[0030]

[0031] in, This represents margin. This represents the candidate target with the best prediction result. Indicates the number of candidate targets;

[0032] In the regression process, the application Calculate the positive target:

[0033]

[0034] in, Indicates the first The last coordinate position of the actual trajectory of each road participant. :

[0035]

[0036] in, express , represent of Norm.

[0037] In the above scheme, the candidate target with the highest confidence score is selected as the prediction target, and the region within a certain range of the prediction target is regarded as the GOI region. The final feature X is obtained by encoding the GOI region, including:

[0038]

[0039] Among them, the The first road participant's Each feature is a traffic feature extracted centered on the road participant, using... Indicates; the The first lane node Each feature value represents an environmental characteristic within the current traffic scenario, used as... Represented by; the weight matrix is ​​represented by express, This represents the layer normalization of the ReLU function. , Indicates belonging to the first The first road participant's coordinates, Indicates the first The first lane node One coordinate; Indicates splicing.

[0040] In the above scheme, the final feature X is used to predict the outcome for each road participant. The predicted trajectories and their corresponding confidence scores are calculated; the classification error is calculated by applying the max-margin loss function. The function calculates the displacement error between all predicted trajectories and their corresponding actual ground trajectory coordinates at each time step, including:

[0041] Use linear layers for each road participant Regression is performed on the predicted trajectories:

[0042]

[0043] in, Indicates the first The road participants in the The first time step The predicted coordinates for each pattern; the classification branch outputs a confidence score for each predicted trajectory:

[0044]

[0045] The regression process, through the application Calculate the displacement error between all predicted trajectories and their corresponding true coordinates at each time step:

[0046]

[0047] In the formula, Indicates the first The road participants in the The actual coordinates of each time step Indicates the total number of time steps;

[0048] The model is trained end-to-end using classification and regression loss functions:

[0049]

[0050] In the formula, Indicates the total loss. Represents classification loss, This represents the regression loss.

[0051] According to a second aspect of the present invention, a vehicle is provided that includes the steps of the target-driven multimodal trajectory prediction method described in any one of the foregoing.

[0052] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory, the memory storing a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the target-driven multimodal trajectory prediction method described in any one of the preceding claims.

[0053] According to a fourth aspect of the present invention, a readable storage medium is provided having a program or instructions stored thereon, which, when executed by a processor, implement the steps of the target-driven multimodal trajectory prediction method described in any one of the preceding claims.

[0054] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0055] (1) This invention introduces target region encoding to encode the scene context and motion trajectory within the range of the predicted target and fuses features within the target of interest (GOI) region to help better understand the target's intent.

[0056] (2) By predicting multiple candidate targets and a set of future trajectories, the present invention reflects different target behaviors or intentions, which helps the system adapt to diverse traffic conditions and improves the flexibility of decision-making.

[0057] (3) By estimating and modeling uncertainty, the present invention enables the system to better handle complex and uncertain traffic conditions, thereby improving the robustness of decision-making. Attached Figure Description

[0058] Figure 1 Here is a flowchart of a target-driven multimodal trajectory prediction method;

[0059] Figure 2 Here is a flowchart of the multimodal trajectory prediction process;

[0060] Figure 3 This is a diagram of the framework structure of a multimodal trajectory prediction network. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0062] The technical problems solved by this invention are mainly three:

[0063] (1) This invention applies U-Net to process vehicle trajectories. The U-Net network retains the contextual information of the trajectory data through its encoder-decoder structure, which can capture global features in the trajectory data, thereby better understanding the overall structure of the trajectory. In addition, this network structure can process features at different scales simultaneously, which is very useful for processing vehicle trajectories, because trajectory data may include information at different scales, such as small changes and large-scale movements of the vehicle. Therefore, using the U-Net network to process vehicle trajectory data can fully utilize its excellent performance in the field of image segmentation, thereby improving the efficiency and performance of trajectory data processing and helping the autonomous driving system to better understand the data.

[0064] (2) This invention generates multiple candidate targets for road participants. In real driving environments, human intentions are highly complex. For example, at an intersection, a driver may have multiple options, such as stopping, accelerating, or turning, and different actions may lead to completely opposite results. This makes understanding their behavioral intentions crucial for future trajectory prediction; by understanding the intentions of others, more reasonable predictions can be made. Therefore, in the target prediction stage, multiple candidate targets (given the final destination expected by the participant within the prediction time window) are predicted for road participants using a Transformer, aiming to address the participant's intention problem as accurately as possible.

[0065] (3) Previous trajectory prediction techniques only focus on the vehicle's historical trajectory and driving environment, mainly concentrating on studying the interaction between vehicles and the interaction between vehicles and the environment within the observation time window. However, they rarely explore the vehicle's driving behavior within the prediction time window, such as the vehicle's evasive behavior at a future moment due to compliance with driving ethics. In order to better understand the vehicle's driving intention, this invention selects the candidate target with the highest confidence score as the prediction target, that is, the destination that the vehicle is most likely to reach, and regards the area within 6 meters of the prediction target as the GOI region. The GOI region is encoded, the road and vehicle features within the range are updated, and the features are aggregated.

[0066] like Figure 1 , Figure 2 and Figure 3 As shown, this invention first performs preprocessing operations on the data, cleaning and processing the raw data, handling missing values, outliers, and noise; performing coordinate system transformation and unit standardization to ensure data consistency; and constructing a lane graph based on a vectorized high-precision map. The first stage involves extracting relevant features from the data: firstly, using U-Net to analyze the historical trajectories of vehicles. Perform feature extraction operations to obtain trajectory features Then, convolution operations are performed using LaneGCN to obtain map features. The second stage applies spatial attention mechanisms to fuse trajectory features. and map features To obtain interaction information between roads and vehicles, and to obtain fused features. The third stage involves candidate target sampling, first using a Transformer to predict targets for each vehicle. Potential candidate targets Apply MLP to each Calculate its confidence score and use The function calculates the error between candidate targets and the true target; finally, it selects the target with the highest confidence score as the predicted target, which is the destination the agent is most likely to reach, and considers the area within 6 meters of the predicted target as the GOI region. Encoding the GOI region yields the final feature X. The fourth stage uses the final agent feature X to predict the target for each vehicle. The predicted trajectories and their corresponding confidence scores are calculated; the classification error is calculated by applying the max-margin loss function. The function calculates the displacement error between all predicted trajectories and their corresponding actual ground trajectory coordinates at each time step.

[0067] like Figure 1 , Figure 2 and Figure 3 As shown, the specific implementation steps of the present invention are as follows:

[0068] S1: Obtain a high-precision autonomous driving dataset and divide it into training, validation and test sets; the dataset contains vehicle trajectory data and lane information from high-precision maps.

[0069] S2: Perform preprocessing operations on the data, clean and process the raw data, handle missing values, outliers and noise; perform coordinate system transformation and unit standardization to ensure data consistency.

[0070] S3: Divide the vehicle trajectory data into observed historical trajectory data. and the trajectory that needs to be predicted (i.e., the actual trajectory), then the U-Net encoder is used to process the historical trajectory data. The data is processed to reduce its spatial dimensionality, and then a decoder is used to map the features extracted by the encoder back to the original data form to obtain the final trajectory features. The collaborative work of the encoder and decoder allows architectures such as U-Net to process multi-scale information and better capture and generate spatiotemporal patterns associated with trajectory data; a lane graph is constructed from a vectorized high-precision map, and convolutional operations are performed on the lane graph using LaneGCN to obtain map features. .

[0071] S4: Considering that the target vehicle's behavior is influenced by surrounding vehicles and the road environment, a broader range of contextual information is needed to improve the realism of the prediction. To achieve this, an interaction fusion module is introduced. This module utilizes LaneGCN and a spatial attention layer to capture interaction information at different levels, including four feature attention modules: A2L (vehicle-to-road), L2L (road-to-road), L2A (road-to-vehicle), and A2A (vehicle-to-vehicle), thus providing the target prediction module with more comprehensive features. .

[0072] S5: In the target prediction network module, predict for each road participant. One potential target (i.e., the final destination expected by a given participant within the forecast time window).

[0073] First, apply Transformer. Generate a set of potential objectives for each road participant:

[0074]

[0075] in, The number of participants in the reasoning Indicates the first The first road participant's A potential target.

[0076] To make the predicted target closer to the real target, this invention designs a classification branch composed of MLP M(·) to output the confidence score corresponding to the potential target:

[0077]

[0078] in, Indicates the first The first road participant's The confidence score of each potential target.

[0079] Furthermore, this invention calculates the relationship between the optimal predicted target and the actual ground target. This is used to further improve the accuracy of predicting the endpoint:

[0080]

[0081] in, This represents margin. This represents the candidate target with the best prediction result. This indicates the number of candidate targets.

[0082] In the regression process, the application Calculate the positive target:

[0083]

[0084] in, Indicates the first The last coordinate position of the actual trajectory of each road participant. :

[0085]

[0086] in, express , represent of Norm.

[0087] Subsequently, this invention selects the target with the highest confidence level as the predicted target, i.e., the destination most likely to be reached by the vehicle, and considers the area within 6 meters of the predicted target as the GOI region. Furthermore, vehicle behavior is closely related to historical trajectories and the driving environment. Previous work has mainly focused on the interaction between agents within the observation time window, neglecting the interaction between the vehicle and the map within the GOI region. Therefore, the GOI region is encoded, and its features are aggregated as follows:

[0088]

[0089] Among them, the The first vehicle The features are traffic features extracted centered on the vehicle, using... Indicates; the The first lane node Each feature value represents an environmental characteristic within the current traffic scenario, used as... The weight matrix is ​​represented by... express; This represents the layer normalization of the ReLU function. , Indicates belonging to the first The first participant ' coordinates, Indicates belonging to the first The first lane node coordinates, Indicates splicing.

[0090] S6: After fusing traffic flow features and environmental features, the final feature Χ is obtained, which is used to predict for each road participant. Each candidate predicted trajectory and its corresponding confidence score.

[0091] Use linear layers for each road participant Regression is performed on the predicted trajectories:

[0092]

[0093] in, Indicates the first The road participants in the The first time step The predicted coordinates for each pattern. The classification branch outputs a confidence score for each predicted trajectory:

[0094]

[0095] The regression process, through the application Calculate the displacement error between all predicted trajectories and their corresponding true coordinates at each time step:

[0096]

[0097] in, Indicates the first The road participants in the The actual coordinates of each time step Indicates the total number of time steps;

[0098] The model is trained end-to-end using classification and regression loss functions:

[0099]

[0100] In the formula, Indicates the total loss. Represents classification loss, This represents the regression loss.

[0101] Finally, after training is complete, the current vehicle's driving trajectory data and lane information from the high-precision map are input, and the candidate predicted trajectory with the highest confidence score is selected as the prediction output.

[0102] The present invention also provides a vehicle comprising the steps of the target-driven multimodal trajectory prediction method described in any one of the foregoing.

[0103] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, implement the steps of the target-driven multimodal trajectory prediction method described above.

[0104] The present invention also provides a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the steps of the target-driven multimodal trajectory prediction method described above.

[0105] In summary, this invention provides a target-driven multimodal trajectory prediction method, vehicle, device, and medium. By more accurately predicting the behavior of other vehicles, pedestrians, and bicycles, the autonomous driving system can better avoid potential dangerous situations, thereby improving road safety and reducing traffic accidents.

[0106] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0108] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target-driven multimodal trajectory prediction method, characterized in that, The method includes: Acquire a high-precision autonomous driving dataset, which includes trajectory data of road participants and vectorized high-precision maps; Historical trajectory data of road participants using U-Net Perform feature extraction operations to obtain trajectory features ; A lane graph is constructed based on a vectorized high-precision map, and convolution operations are performed on the lane graph using LaneGCN to obtain map features. ; Fusing trajectory features using LaneGCN and spatial attention layer and map features To capture all interaction information between road participants and roads, and obtain fused features. ; Based on the fused features Predictions for each road participant candidate targets And apply MLP to each candidate target Calculate the confidence score and use The function calculates the error between the best candidate target and the true target. The candidate target with the highest confidence score is selected as the prediction target, and the region within a certain range of the prediction target is regarded as the GOI region. The GOI region is encoded to obtain the final feature X. Using the final feature X, predict the outcome for each road participant. The predicted trajectories and their corresponding confidence scores are calculated; the classification error is calculated by applying the max-margin loss function. The function calculates the displacement error between all predicted trajectories and their corresponding actual ground trajectory coordinates at each time step. After training, the predicted trajectory with the highest confidence score is selected as the final trajectory prediction result.

2. The target-driven multimodal trajectory prediction method according to claim 1, characterized in that, The method also includes: The data is preprocessed to clean and process the raw data, handle missing values, outliers and noise, and perform coordinate system transformation and unit standardization.

3. The target-driven multimodal trajectory prediction method according to claim 1, characterized in that, Historical trajectory data of road participants using U-Net Perform feature extraction operations to obtain trajectory features ,include: The trajectory data of road participants is divided into observed historical trajectory data. and the actual trajectory data that needs to be predicted Using U-Net encoders to process historical trajectory data Feature extraction is performed, and then the U-Net decoder is used to map the features extracted by the encoder back to the original data form to obtain trajectory features. .

4. The target-driven multimodal trajectory prediction method according to claim 1, characterized in that, All interaction information between road participants and roads includes: road participant to road A2L, road to road L2L, road to road participant L2A, and road participant to road participant A2A.

5. The target-driven multimodal trajectory prediction method according to claim 1, characterized in that, Based on the fused features Predictions for each road participant candidate targets And apply MLP to each candidate target Calculate the confidence score and use The function calculates the error between the best candidate target and the true target, including: Applying Transformer Generate a set of candidate targets for each road participant: in, G For the candidate target set, For the number of road participants, Indicates the first The first road participant's One candidate target; To make candidate targets closer to the real target, a method based on MLP is designed. M The classification branch formed by (·) is used to output the confidence score corresponding to each candidate target: in, For the set of confidence scores, Indicates the first The first road participant's Confidence scores for each candidate target; Calculate the ratio between the best candidate target and the true target This is used to further improve the accuracy of target prediction. in, This represents margin. This represents the candidate target with the best prediction result. Indicates the number of candidate targets; In the regression process, the application Calculate the positive target: in, Indicates the first The last coordinate position of the actual trajectory of each road participant. : in, express , represent of Norm.

6. The target-driven multimodal trajectory prediction method according to claim 1, characterized in that, The candidate target with the highest confidence score is selected as the prediction target, and the region within a certain range of the prediction target is regarded as the GOI region. The GOI region is encoded to obtain the final feature X, including: Among them, the The first road participant's Each feature is a traffic feature extracted centered on the road participant, using... Indicates; the The first lane node Each feature value represents an environmental characteristic within the current traffic scenario, used as... Represented by; the weight matrix is ​​represented by express, This represents the layer normalization of the ReLU function. , Indicates belonging to the first The first road participant's coordinates, Indicates the first The first lane node One coordinate; Indicates splicing.

7. The target-driven multimodal trajectory prediction method according to claim 6, characterized in that, Using the final feature X, predict the outcome for each road participant. The predicted trajectories and their corresponding confidence scores are calculated; the classification error is calculated by applying the max-margin loss function. The function calculates the displacement error between all predicted trajectories and their corresponding actual ground trajectory coordinates at each time step, including: Use linear layers for each road participant Regression is performed on the predicted trajectories: in, Indicates the first The road participants in the The first time step The predicted coordinates for each pattern; the classification branch outputs a confidence score for each predicted trajectory: The regression process, through the application Calculate the displacement error between all predicted trajectories and their corresponding true coordinates at each time step: In the formula, Indicates the first The road participants in the The actual coordinates of each time step Indicates the total number of time steps; The model is trained end-to-end using classification and regression loss functions: In the formula, Indicates the total loss. Represents classification loss. This represents the regression loss.

8. A vehicle, characterized in that, The vehicle includes the steps of the target-driven multimodal trajectory prediction method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, implement the steps of the target-driven multimodal trajectory prediction method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, It stores a program or instructions that, when executed by a processor, implement the steps of the target-driven multimodal trajectory prediction method according to any one of claims 1 to 7.

Images