A graph-centric vehicle multimodal trajectory prediction method

Abstract

The present invention belongs to the field of autonomous driving technology and discloses a graph-centric vehicle multimodal trajectory prediction method, comprising: constructing a local graph for each traffic-participating vehicle; splicing the geometric features, type features, and dynamic interaction relationships between lane segments and traffic-participating vehicles, and then performing feature extraction to obtain node features of the local graph; projecting all encoded local graphs onto a global lane graph to obtain a global graph; aggregating the node features of all local graphs, and then performing a lane convolution operation to obtain a local feature fusion vector; performing feature extraction on the local feature fusion vector to obtain a global interaction feature vector; weightedly combining the local feature fusion vector and the global interaction feature vector to obtain node features of the global graph; projecting the node features of the global graph back to the local graph to obtain an enhanced local graph; and obtaining predicted trajectories of each traffic-participating vehicle and the confidence level of each predicted trajectory based on the enhanced local graph.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and in particular relates to a graph-centric vehicle multimodal trajectory prediction method. Background Art

[0002] With the development of intelligent transportation systems, the use of autonomous vehicles (AVs) in complex traffic environments is increasing. Accurate trajectory prediction is a key technology for ensuring the safety and efficiency of autonomous driving. The goal of trajectory prediction is to predict the likely future trajectories of traffic participants (such as vehicles and pedestrians) based on their historical motion data and environmental information. This process is crucial for implementing autonomous driving functions such as decision-making, path planning, and obstacle avoidance.

[0003] Existing trajectory prediction methods can be roughly divided into three categories: models based on traditional methods, models based on deep learning, and models based on reinforcement learning. Traditional methods generally rely on physical models or rule-based approaches to predict trajectories. These methods are often limited to simple assumptions and have difficulty handling dynamic and complex traffic scenarios. In recent years, deep learning methods, particularly trajectory prediction models based on technologies such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs), have made significant progress. By learning from historical trajectories and environmental information, these models can better capture the motion patterns of intelligent agents and complex traffic interactions.

[0004] However, existing deep learning methods often suffer from several shortcomings. First, most models represent the historical trajectories of traffic participants and environmental information separately as single vectors or feature vectors. This approach ignores the interactions between different traffic participants and the influence of environmental information (such as lanes and traffic lights). Second, when dealing with complex scenarios, traditional models fail to effectively utilize the spatial and topological structures in traffic scenes, especially in scenes with complex traffic rules such as intersections and roundabouts. Finally, existing models often lack flexible global interaction mechanisms, making it difficult to accurately capture long-range interactions between agents, thus affecting the accuracy and reliability of trajectory prediction. Summary of the Invention

[0005] The purpose of the present invention is to provide a graph-centric vehicle multimodal trajectory prediction method, which overcomes the problems of low spatial interaction ability and poor integration with lane environment in existing methods. It can capture multiple target positions of traffic participating vehicles in complex road environments and ultimately output the vehicle multimodal predicted trajectory.

[0006] The technical solution provided by the present invention is:

[0007] A graph-centric vehicle multimodal trajectory prediction method, comprising:

[0008] Obtaining historical trajectories of vehicles participating in the traffic and lane lines corresponding to the historical trajectories, and dividing the lane lines corresponding to the historical trajectories into lane segments;

[0009] Using the center point of the lane segment as a node and the positional relationship between the center points of adjacent lane segments as an edge, a local graph of each participating vehicle is constructed;

[0010] After combining the geometric features and type features of lane segments and the dynamic interaction relationship between lane segments and participating vehicles, feature extraction is performed to obtain node features of the local graph.

[0011] Construct a global lane map containing all lanes in the scene, encode the local map of each participating vehicle separately, and project all the encoded local maps onto the global lane map to obtain a global map;

[0012] After aggregating the node features of all local graphs, lane convolution is performed on the aggregated feature vector to obtain a local feature fusion vector;

[0013] Performing feature extraction on the local feature fusion vector through a global interaction relationship encoding layer to obtain a global interaction feature vector;

[0014] Weightedly combining the local feature fusion vector and the global interaction feature vector to obtain node features of the global graph;

[0015] Projecting the node features of the global graph back to the local graph of each traffic participating vehicle to obtain an enhanced local graph;

[0016] The predicted trajectory of each traffic participant vehicle and the confidence level of each predicted trajectory are obtained according to the enhanced local graph.

[0017] Preferably, the method for encoding the partial image of each traffic participant vehicle is:

[0018] A lane convolution operation is performed on the node features of the local graph, and an average pooling operation is performed after each layer of convolution, and the results of the convolution operation and the average pooling operation are added together to obtain an encoded local graph node feature vector.

[0019] Preferably, the global interaction relationship encoding layer is composed of a multi-head attention layer.

[0020] Preferably, obtaining the predicted trajectory of each traffic participant vehicle and the confidence level of each predicted trajectory based on the enhanced local graph includes the following steps:

[0021] Step 1: Use the first prediction unit to predict each node in the enhanced local graph, determine the probability of the node being the target point and the position residual from the node to the true target point;

[0022] Step 2: Select the K nodes with the highest probability of being the target point as candidate target points, and pass the candidate target points through the multi-head attention layer to perform information interaction to obtain interaction features; input the interaction features into the second prediction unit to obtain the probability of the candidate target point being the target point and the position residual from the candidate target point to the true target point;

[0023] Step 3: Use the Bezier curve to fit the trajectory from the current coordinate point to the candidate target point to obtain the reference trajectory;

[0024] Step 4: Fine-tune the reference trajectory to obtain a predicted trajectory, and use the probability of the candidate target point being the target point as the confidence of the corresponding predicted trajectory.

[0025] Preferably, the first prediction unit and the second prediction unit both adopt multi-layer perceptrons.

[0026] Preferably, in step 4, the method for fine-tuning the reference trajectory to obtain the predicted trajectory is:

[0027] The vehicle's position at multiple future moments is obtained through the reference trajectory, the offset at each moment is determined through a multi-layer perceptron, and the position at multiple future moments is fine-tuned according to the offset to obtain a predicted trajectory.

[0028] The beneficial effects of the present invention are:

[0029] The graph-centric vehicle multimodal trajectory prediction method provided by the present invention fully utilizes the lane environment around the intelligent agent and can well capture the relationship between the intelligent agent and the lane, thereby accurately predicting the future trajectory of the intelligent agent in a complex traffic environment.

[0030] By combining lane convolution with a multi-head attention layer, the present invention can more comprehensively model the complex relationships between vehicles involved in traffic and improve the accuracy of trajectory prediction. It also performs a weighted fusion of the features output by LaneGCN and the attention mechanism, and determines the weights through a learning method, so that the model can adjust the degree of attention to local and global features according to the dynamics of the actual scene, thereby enhancing the robustness of the prediction model.

[0031] The present invention combines the local graph of the intelligent agent for optimization in the prediction head part, which can make the predicted trajectory more consistent with the road characteristics and make the prediction more accurate and reasonable.

[0032] The present invention adopts a two-stage decoder, which predicts target points and trajectories in stages through "coarse screening + fine tuning", effectively improving the accuracy, diversity, stability and expressiveness of the prediction, and enabling the decoder to have stronger structured modeling capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of the graph-centric vehicle multimodal trajectory prediction method described in the present invention.

[0034] Figure 2 This is a framework diagram of the multi-vehicle joint trajectory prediction model described in the present invention. DETAILED DESCRIPTION

[0035] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0036] like Figure 1-2 As shown, the present invention provides a graph-centric vehicle multimodal trajectory prediction method, and the specific implementation process is as follows.

[0037] 1. Build and train a vehicle multimodal trajectory prediction model

[0038] The vehicle multimodal trajectory prediction model includes: Encoder, global interaction module and decoder module.

[0039] The encoder includes a multi-layer perceptron and a lane convolution layer. The global interaction module includes a multi-layer perceptron , Multilayer Perceptron Lane convolution layer and multi-head attention layer. The decoder module includes the first stage prediction head and the second stage prediction head; the first stage prediction head adopts a multi-layer perceptron The second stage prediction head consists of a multi-head attention layer, a multi-layer perceptron and multilayer perceptrons .

[0040] The specific training process is as follows:

[0041] 1. Scene initialization

[0042] The first The past motion of the participating vehicles is represented as a set of two-dimensional points , encoded in the past The center position of the traffic participating vehicles within a time step, that is:

[0043] ;

[0044] in, In the past Moment The absolute horizontal and vertical coordinates of the center point of each traffic vehicle in the map coordinate system, superscript The representative coordinates are historical trajectories.

[0045] At the same time, in the scene The lane lines are also represented as a set of two-dimensional points , the center line of each lane is fixedly divided into The center line of the lane line is generally divided into segments at 2m intervals. The center point coordinates of the divided lane segment are used as the position coordinates of the lane segment, encoding the lane features near the agent:

[0046] ;

[0047] in, Indicates the Lane line The absolute horizontal and vertical coordinates of the center point of each lane segment in the map coordinate system.

[0048] The goal of this invention is to predict the coordinate points of the possible driving trajectories of N participating vehicles in the scene within T time steps in the future:

[0049] ;

[0050] in, In the future Moment The absolute horizontal and vertical coordinates of the center point of each traffic vehicle in the map coordinate system, superscript The representative coordinates are the predicted trajectories.

[0051] 2. Local map Construction

[0052] Obtain all lanes that traffic participating vehicles may reach within the prediction time T through the existing dataset , and will also observe all lanes that the vehicle has passed through within the historical time L The lanes are extracted and then divided into multiple continuous lane segments with a 2m interval according to the lane centerline. The center point of each lane segment is used as the Node representation , these nodes are structurally connected to each other, forming the following directed graph:

[0053] ;

[0054] Among them, the local map Each node in Indicates the The first vehicle participating in the traffic The position of each lane segment in the local map, The topological structure of the lane encodes the structural relationship between the predecessor, successor, left neighbor and right neighbor, which are represented by four types of connecting edges. Indicates the number of nodes in each local graph.

[0055] For each node The embedded feature vector representation of :

[0056] ;

[0057] in, It represents the geometric characteristics of the lane segment (node), It represents the lane segment type feature. It represents the dynamic interaction relationship between the lane segment and the agent, and C represents the node embedding feature dimension.

[0058] In this embodiment, set ;

[0059] In the above formula, Indicates the center point coordinates of the lane segment centerline, Indicates the direction of the lane segment, In other embodiments, some road coefficients of the lane segment, such as the friction coefficient, can also be considered. If the factors are added, The geometric feature dimension changes, Represents the geometric feature dimension.

[0060] For lane segment type characteristics For example:

[0061] ;

[0062] In the above formula, Indicates the lane end category, which includes straight lane, turning lane, fork and ramp; After encoding, they correspond to the vectors [1,0,0,0], [0,1,0,0], [0,0,1,0], [0,0,0,1] respectively. Indicates whether the lane section is controlled by a red light. Indicates that the lane section is controlled by a red light; Indicates whether the lane segment is the target lane. Indicates that the lane segment is the target lane of the agent. S represents the type feature dimension.

[0063] for For example, the past movement of the traffic participating vehicles can be represented as a set of coordinate points , which defines the center position of the vehicle at consecutive time steps. Therefore, these coordinate points Convert to displacement sequence , as follows, and Projected to local graph nodes coordinate system in order to encode the participant's movement information in relation to the map.

[0064] ;

[0065] ;

[0066] In the above formula, Indicates that the vehicle is The displacement of time, Represents the coordinate information of the displacement sequence in the Frenet coordinate system of the lane segment, Represents the Frenet projection operation of the local graph node in the Cartesian coordinate system. At the same time, it stipulates that when the distance between the traffic participating vehicle and the lane segment exceeds 5 meters, the intelligent body motion embedding of the node is replaced by an all-zero vector .

[0067] 3. LaneG encoder (modeling of traffic vehicle time series)

[0068] The three features of the node are embedded and spliced ​​together, and then projected into the final node feature through the multi-layer perceptron (MLP) in the LaneG encoder:

[0069] ;

[0070] In summary, the local graph of the intelligent agent Depend on and Together they constitute: , indicating the The graph structure of the traffic participating vehicles encodes the trajectory information of the traffic participating vehicles and the surrounding environment information related to the prediction. Represents the node embedding feature vector A collection of yes The number of nodes in the.

[0071] For each constructed local graph of the agent , apply the lane convolution layer in the LaneG encoder to transfer information with surrounding nodes to obtain the updated node embedding , as follows:

[0072] ;

[0073] In the above formula, and are all learnable parameters. is and The nonlinear layer consists of a summation operation on all relations m and hop number n. The hop number n represents the distance between nodes. This multi-hop operation simulates dilated convolution, effectively expanding the receptive field. D represents the embedding feature dimension of each local graph node after convolution.

[0074] At the same time, in order to solve the long-term dependency, a graphics shortcut layer is added after each lane convolution. After each lane convolution, an average pooling operation is added to aggregate all node features into a global embedding and pass it directly to the graph. Node embedding In the following formula:

[0075] ;

[0076] ;

[0077] In the above formula, represents the global embedding after aggregation, Representation diagram The number of nodes, is the final node representation after lane convolution and average pooling in the encoder and addition.

[0078] The role of the encoder is to model the historical trajectory of the agent in time series. The nodes in the encoded graph structure can contain historical trajectory information, realizing the interaction of the agent in time series.

[0079] 4. Global interaction (spatial interaction modeling between vehicles involved in traffic)

[0080] For each agent After encoding, we get a picture that integrates the surrounding environment information. , and then construct a global lane graph containing all lanes in the scene , global lane graph The nodes are all pooled by lane pooling operation Projection to To construct, the specific operations are as follows:

[0081] 4.1 Construction of global graph nodes and connection edge structure:

[0082] ;

[0083] Where N represents the local graph Quantity, that is, the number of traffic participating vehicles in the scene, and M represents the number of nodes in the global graph.

[0084] At the same time, the edge relationship matrix Determined by the topology of the lane and related to each local graph Connection relationship Stay consistent.

[0085] 4.2 Node Feature Embedding Build:

[0086] For each node in the global graph , from all containing in picture Search all Related node collection ,for have:

[0087] ;

[0088] in, It is a local graph Node, Representation node and True distance, Indicates the distance threshold of neighbor nodes, usually 5m.

[0089] For all local neighbor nodes Node characteristics Perform aggregation to generate global nodes Features :

[0090] ;

[0091] in, Represents a multi-layer perceptron (MLP) for processing node features and relative position information to generate enhanced node-position feature representation. Represents the splicing of local node features and relative positions; It is also a multi-layer perceptron, which is used to further globally process the information after the aggregation of neighbor features to generate the final feature representation of the target node. E represents the embedding feature dimension of the node in the global graph. express The set of neighbor nodes.

[0092] Finally, the aggregated features As a global graph node Feature embedding of: , .

[0093] 4.3 Information transmission of global graph:

[0094] Convolution operations can quickly extract effective local structural information, such as the movement of participants within the current lane and local obstacles. However, they cannot effectively capture global scene features when comparing long-term and long-distance contextual features. Unlike convolution, the core of the attention mechanism is to assign different weights to different positions, allowing the model to selectively focus on important information as needed, especially global dependencies in long-term and long-distance interactions. Therefore, the attention mechanism enables the model to learn and capture the interactive relationships between multiple targets and multiple time steps, and is particularly suitable for modeling long-range dependencies and complex interactions. Therefore, in the global interaction process, based on the use of lane convolution operations, the self-attention mechanism is introduced to further pass messages to the information generated by convolution, thereby strengthening global information.

[0095] In the global graph, the lane convolution layer in the global interaction module is first used to fuse local features between nodes to capture the local context representation of the nodes:

[0096] ;

[0097] Then update the node features The input is sent to the global interaction relationship encoding layer composed of the multi-head attention layer in the global interaction module, and the spatial interaction relationship between vehicles and vehicles and vehicles and roads is captured by the multi-head self-attention MHSA, which is recorded as the global interaction feature vector :

[0098] Among them, PE represents the position feature vector of a node.

[0099] Finally, the final node embedding of the global graph is the combination of the vectors obtained by the two operations. The combination method is to splice the two vectors together and use a fully connected layer to generate attention weights. and , as the combined weight, as follows:

[0100] ;

[0101] ;

[0102] in, and represents the combined weight of the two vectors, Q represents the node embedding feature dimension after the lane convolution layer and the multi-head attention layer, Represents the fused feature vector after global interaction.

[0103] In summary, the feature vector obtained by lane convolution and multi-head attention layer and weighted combination of the two is As the node feature vector of the final global graph, this feature vector highly integrates local lane features and global context features, and well realizes the interaction between agents and between agents and lanes. The final global graph is represented as .

[0104] 5. Lane trajectory decoder

[0105] 5.1 Multimodal trajectory prediction for target agents is based on the local graph of each agent. Therefore, the node features obtained in the global graph must be projected back to the local graph. To enhance the local image The feature representation of the graph contains contextual information from global interactions.

[0106] This part is the reverse process of 4.2 and will not be described here.

[0107] Finally, each local graph The node features are newly represented .

[0108] 5.2 Prediction Head Composition

[0109] 5.2.1. First prediction head: using a simple multi-layer perceptron To local map Each node in is predicted, and the probability of the predicted node as the target point and the position residual of the node to the true target point are:

[0110] ;

[0111] in, Represents a two-layer multilayer perceptron, whose activation function is ; express The probability set of the middle node being the target point; Represents the position residual set from the node to the true target point, where Contains Relative position deviation and orientation residual.

[0112] 5.2.2. Second prediction head: According to The K nodes with the highest scores are selected as candidate target points TOP-K by the size of the probability, and then the feature set of the candidate target points is , after a streamlined multi-head attention layer, information interaction is performed between candidate points to obtain interactive features :

[0113] ;

[0114] ;

[0115] Through multi-layer perceptron Finally, the interaction features are classified and scored and the position residuals are estimated:

[0116] ;

[0117] in, express The probability set of candidate target points as target points; Represents the set of position residuals from candidate target points to true target points.

[0118] 5.2.3 Bezier curve fitting:

[0119] The trajectory curve between the target point and the initial position of the vehicle is fitted, and the trajectory from the current coordinate point to the target point is fitted using the Bezier curve. The final position is the predicted target point coordinate. and the vehicle's position coordinates at the predicted time , and assuming that the vehicle moves in a constant acceleration kinematic model in a short period of time, the final trajectory curve (in the Frenet coordinate system) is:

[0120] ;

[0121] in, ;

[0122] Assuming that the vehicle is moving with uniform acceleration, the vehicle position at each moment can be calculated :

[0123] ;

[0124] ;

[0125] ;

[0126] 5.2.3.2 Trajectory Optimization:

[0127] Since the movement of the vehicle is not always uniformly accelerated, we can use the Bezier curve as a reference trajectory to predict the position of the vehicle in the next 60 frames, that is, the future trajectory of the vehicle, and use a multi-layer perceptron to calculate the vehicle's position. Regression fine-tunes the offset to achieve accurate predictions.

[0128] ;

[0129] in, , Indicates the driving direction offset and vertical direction offset of the vehicle trajectory in the next T frames, where T represents the number of frames of the predicted future trajectory. Represents the projection to after the global interaction module Node feature representation of graphs. Indicates the application of multi-layer perceptron.

[0130] The final fine-tuned trajectory points are:

[0131] ;

[0132] in, Indicates the initial trajectory The point of time, express The direction of the tangent vector at time express Normal vector direction at this moment.

[0133] Finally, each model The graph outputs K multimodal trajectories and their confidence scores:

[0134] ;

[0135] ;

[0136] Among them, each trajectory is a time series, representing the future position. Indicates the predicted trajectories, which contain the vehicle's future T-frame position information, . Represents the set of its corresponding confidence scores.

[0137] The entire model outputs the future trajectories of N traffic participating vehicles and their corresponding confidence functions as follows:

[0138] ;

[0139] ;

[0140] 6. Loss Function

[0141] The model is trained by reducing the error between the predicted value and the true value, and updating the parameters in the model network by backpropagating the loss, guiding the model to perform end-to-end optimization, so that the trajectory predicted by the prediction model is closer to the true value.

[0142] The hyperparameters of each module in the multi-vehicle joint trajectory prediction model are shown in Table 1.

[0143] Table 1 Hyperparameters of the multi-vehicle joint trajectory prediction model structure

[0144]

[0145] The loss function used in the training of the multi-vehicle joint trajectory prediction model is:

[0146] ;

[0147] As a preference, set , , , ;

[0148] in, Represents the model classification loss. Each node is judged to be the final target point using cross entropy loss:

[0149] ;

[0150] in, N is the total number of nodes, Indicates the Whether the node is the target point. is the probability that a node is predicted to be a target.

[0151] Represents the residual loss between each node and the end point (target point), and the loss is expressed as:

[0152] ;

[0153] ;

[0154] ;

[0155] in, Indicates the position and orientation difference between the candidate point and the true target point.

[0156] Represents the trajectory regression loss. After the candidate target point is refined, K trajectories are output. The one closest to the GT is selected to calculate the trajectory regression loss. The loss is expressed as:

[0157] ;

[0158] in, T Indicates the number of time steps for trajectory prediction Indicates the predicted Trajectories at time step The position coordinates of represents the one closest to the true trajectory, represents the true trajectory at time step The location coordinates of Represents the L1 norm, which is the sum of the absolute values ​​of the main elements of the position coordinates.

[0159] It represents the loss of fine-tuning the trajectory point under Frenet, and the loss value of correcting the vehicle trajectory at each time step. The loss is expressed as:

[0160] ;

[0161] in, represents the trajectory point after fine-tuning, represents the true trajectory point, T Represents the number of frames (time steps) for predicting the future trajectory of the vehicle.

[0162] Represents the multimodal trajectory probability NLL loss (top-k trajectories + confidence), modeling the probability of the target trajectory appearing in multiple predicted trajectories. The loss value is:

[0163] ;

[0164] After training, the multi-vehicle joint trajectory prediction model can be applied to the trajectory prediction of traffic participating vehicles in actual driving scenarios.

[0165] 2. Use the trained multi-vehicle joint trajectory prediction model to predict multi-vehicle joint trajectories

[0166] Except for the input data processing, the prediction process is basically the same as the training process.

[0167] Data processing

[0168] The input data is mainly obtained by vehicle-mounted sensors, using The data acquisition system is equipped with lidar, millimeter-to-millimeter radar, depth camera, GPS positioning and high-precision map to obtain traffic environment information of surrounding vehicles and obstacles. Then, after target detection and classification (YOLOv8), obstacle detection and tracking, lane detection (LaneATT) and traffic signal recognition (YOLOv5 + Traffic light head), the ICP algorithm is used to spatially align the objects in the lidar point cloud data and camera image to obtain high-precision position information, thereby obtaining complete and accurate historical trajectory information of the vehicle and participating vehicles in the traffic, as well as the lane line position information in the traffic scenario. Finally, the vehicle status and historical trajectory, the historical trajectory of other vehicles, the lane line topology, and the lane line position information are output.

[0169] Building a graph structure

[0170] The historical trajectory of each traffic participating vehicle and the position information of each lane line around it are expressed as follows:

[0171] ;

[0172] ;

[0173] Based on the current position and scene information of the traffic participating vehicle to be predicted, the lane segments passed by the traffic participating vehicle in the past time range T are generally selected to be extracted. The lane segments that the traffic participating vehicle may pass through in the next 6 seconds (60 frames) are extracted, and the lane lines are divided into continuous lane segments at intervals of 2m along the center line.

[0174] The center point of each lane segment is used as the node of the local graph to form a node matrix The topological structure in the real scene is the connection edge relationship between nodes, which has four types: left neighbor, right neighbor, predecessor, and successor, forming an edge relationship matrix. The characteristic information variables of each node are used as the node's embedding vector to form the node's embedding vector matrix The final local graph is represented as , as the graph structure input information of the subsequent prediction model.

[0175] For each node The embedded feature vector representation of :

[0176] ;

[0177] in, It represents the geometric characteristics of the lane segment (node), It represents the lane segment type feature. It represents the dynamic interaction relationship between the lane segment and the agent, and C represents the node embedding feature dimension.

[0178] set up ;

[0179] In the above formula, Indicates the center point coordinates of the lane segment centerline, Indicates the direction of the lane segment, Indicates the curvature of the lane segment. For example:

[0180] ;

[0181] In the above formula, Indicates the lane end category, which includes straight lane, turning lane, fork and ramp; After encoding, they correspond to the vectors [1,0,0,0], [0,1,0,0], [0,0,1,0], [0,0,0,1] respectively. Indicates whether the lane section is controlled by a red light. Indicates that the lane section is controlled by a red light; Indicates whether the lane segment is the target lane. Indicates that the lane segment is the target lane of the agent. S represents the type feature dimension.

[0182] for For example, the past movement of the traffic participating vehicles can be represented as a set of coordinate points That is, it defines the center position of the vehicle at consecutive time steps. Therefore, these coordinate points Convert to displacement sequence , as follows, and Projected to local graph nodes coordinate system in order to encode the participant's movement information in relation to the map.

[0183] ;

[0184] ;

[0185] In the above formula, Indicates that the vehicle is The displacement of time, Represents the displacement sequence in the lane segment Coordinate information under the coordinate system, Represents the coordinate system from the Cartesian coordinate system to the local graph node The projection operation under the same regulation is that when the distance between the traffic participating vehicle and the lane segment exceeds 5 meters, the intelligent body motion embedding of the node is replaced by a full 0 vector. .

[0186] 3. encoder

[0187] The three features of the node are embedded and spliced ​​together. The multi-layer perceptron (MLP) in the encoder projects the final node features:

[0188] ;

[0189] In summary, the local graph of the intelligent agent is and Together they constitute: , indicating the The graph structure of the traffic participating vehicles encodes the trajectory information of the traffic participating vehicles and the surrounding environment information related to the prediction. Represents the node embedding feature vector A collection of yes The number of nodes in the.

[0190] For each constructed local graph of the agent ,application The lane convolution layer in the encoder transfers information with surrounding nodes to obtain updated node embeddings , as follows:

[0191] ;

[0192] In the above formula, and are all learnable parameters. is and The nonlinear layer consists of a summation operation on all relations m and hop number n. The hop number n represents the distance between nodes. This multi-hop operation simulates dilated convolution, effectively expanding the receptive field. D represents the embedding feature dimension of each local graph node after convolution.

[0193] After each lane convolution, a graphics shortcut layer is added. After each lane convolution, an average pooling operation is added to aggregate all node features into a global embedding and pass it directly to the graph. Node embedding In the following formula:

[0194] ;

[0195] ;

[0196] In the above formula, represents the global embedding after aggregation, Representation diagram The number of nodes, is the final node representation after lane convolution and average pooling in the encoder and addition.

[0197] 4. Global interaction (spatial interaction modeling between vehicles involved in traffic)

[0198] For each agent After encoding, we get a picture that integrates the surrounding environment information. Then, a global lane graph G containing all lanes in the scene is constructed. The nodes of the global lane graph G are all lanes in the scene through lane pooling operation. Projection to To construct, the specific operations are as follows:

[0199] 4.1 Construction of global graph nodes and connection edge structure:

[0200] ;

[0201] Where N represents the local graph Quantity, that is, the number of traffic participating vehicles in the scene, and M represents the number of nodes in the global graph.

[0202] At the same time, the edge relationship matrix Determined by the topology of the lane and related to each local graph Connection relationship Stay consistent.

[0203] 4.2 Construction of node feature embedding F:

[0204] For each node in the global graph , from all containing in picture Search all Related node collection ,for have: .

[0205] in, It is a local graph Node, Representation node and True distance, Indicates the distance threshold of neighbor nodes, usually 5m.

[0206] For all local neighbor nodes Node characteristics Perform aggregation to generate global nodes Features :

[0207] ;

[0208] in, Represents a multi-layer perceptron (MLP) for processing node features and relative position information to generate enhanced node-position feature representation. Represents the splicing of local node features and relative positions; It is also a multi-layer perceptron, which is used to further perform global processing on the information after the aggregation of neighbor features to generate the final feature representation of the target node. E represents the embedding feature dimension of the node in the global graph.

[0209] Finally, the aggregated features As a global graph node Feature embedding of: .

[0210] 4.3 Information transmission of global graph:

[0211] Convolution operations can quickly extract effective local structural information, such as the movement of participants within the current lane and local obstacles. However, they cannot effectively capture global scene features when comparing long-term and long-distance contextual features. Unlike convolution, the core of the attention mechanism is to assign different weights to different positions, allowing the model to selectively focus on important information as needed, especially global dependencies in long-term and long-distance interactions. Therefore, the attention mechanism enables the model to learn and capture the interactive relationships between multiple targets and multiple time steps, and is particularly suitable for modeling long-range dependencies and complex interactions. Therefore, in the global interaction process, based on the use of graph convolution operations, the self-attention mechanism is introduced to further pass messages to the information generated by convolution, thereby strengthening global information.

[0212] In the global graph, the lane convolution layer in the global interaction module is first used to fuse local features between nodes to capture the local context representation of the nodes:

[0213] ;

[0214] Then update the node features The input is sent to the global interaction relationship encoding layer composed of the multi-head attention layer in the global interaction module. The spatial interaction relationship between vehicles and vehicles and vehicles and roads is captured by multi-head self-attention MHSA, which is recorded as the global interaction feature vector:

[0215] ;

[0216] Among them, PE represents the position feature vector of a node.

[0217] Finally, the final node embedding of the global graph is the combination of the vectors obtained by the two operations. The combination method is to splice the two vectors together and use a fully connected layer to generate attention weights. , , as the combined weight, as follows:

[0218] ;

[0219] ;

[0220] in, , represents the combined weight of the two vectors, Q represents the node embedding feature dimension after the lane convolution layer and the multi-head attention layer, Represents the fused feature vector after global interaction.

[0221] In summary, the feature vector obtained by lane convolution and multi-head attention layer and weighted combination of the two is As the node feature vector of the final global graph, this feature vector highly integrates local lane features and global context features, and well realizes the interaction between agents and between agents and lanes. The final global graph is represented as .

[0222] 5. Lane trajectory decoder

[0223] 5.1 Multimodal trajectory prediction for target agents is based on the local graph of each agent. Therefore, the node features obtained in the global graph must be projected back to the local graph. To enhance the local image The feature representation of the graph contains contextual information from global interactions.

[0224] Finally, each local graph The node features are newly represented .

[0225] 5.2 Prediction Head Composition

[0226] 5.2.1. First Prediction Head: Using Multilayer Perceptron To local map Each node in is predicted, and the probability of the predicted node as the target point and the position residual of the node to the true target point are:

[0227] ;

[0228] in, Represents a two-layer multilayer perceptron, whose activation function is ; express The probability set of the middle node being the target point; Represents the position residual set from the node to the true target point, where Contains Relative position deviation and orientation residual.

[0229] 5.2.2. Second prediction head: According to The K nodes with the highest scores are selected as candidate target points TOP-K by the size of the probability, and then the feature set of the candidate target points is , after a streamlined multi-head attention layer, information interaction is performed between candidate points to obtain interactive features ,

[0230] ;

[0231] ;

[0232] Through multi-layer perceptron Finally, the interaction features are classified and scored and the position residuals are estimated:

[0233] ;

[0234] in, express The probability set of candidate target points as target points; Represents the set of position residuals from candidate target points to true target points.

[0235] 5.2.3 Bezier curve fitting:

[0236] The trajectory curve between the target point and the initial position of the vehicle is fitted, and the trajectory from the current coordinate point to the target point is fitted using the Bezier curve. The final position is the predicted target point coordinate. and the vehicle's position coordinates at the predicted time , and assuming that the vehicle moves in a constant acceleration kinematic model in a short period of time, the final trajectory curve (in the Frenet coordinate system) is:

[0237]

[0238] in, .

[0239] Assuming that the vehicle is moving with uniform acceleration, the vehicle position at each moment can be calculated :

[0240] ;

[0241] ;

[0242] ;

[0243] 5.2.3.2 Trajectory Optimization:

[0244] Since the movement of the vehicle is not always uniformly accelerated, we can use the Bezier curve as a reference trajectory to predict the position of the vehicle in the next 60 frames, that is, the future trajectory of the vehicle, and use a multi-layer perceptron to calculate the vehicle's position. Regression fine-tunes the offset to achieve accurate predictions.

[0245] ;

[0246] in, Indicates the driving direction offset and vertical direction offset of the vehicle trajectory in the next T frames, where T represents the number of frames of the predicted future trajectory. Represents the projection to after the global interaction module Node feature representation of graphs. Indicates the application of multi-layer perceptron.

[0247] The final fine-tuned trajectory points are:

[0248]

[0249] in, Indicates the initial trajectory The point of time, express The direction of the tangent vector at time express Normal vector direction at this moment.

[0250] Finally, each model The graph outputs K multimodal trajectories and their confidence scores:

[0251] ;

[0252] ;

[0253] Among them, each trajectory is a time series, representing the future position. Indicates the predicted trajectories, which contain the vehicle's future T-frame position information, . Represents the set of its corresponding confidence scores.

[0254] The entire model outputs the future trajectories of N traffic participating vehicles and their corresponding confidence functions as follows:

[0255] .

[0256] The graph-centric, multimodal trajectory prediction method for multiple vehicles proposed in this paper models intelligent agents in a distributed and map-aware manner by constructing a local lane graph to represent the contextual information (surrounding environment information) and historical trajectory data of traffic participants. During the local graph construction process, the map's topological structure is preserved and fine-grained environmental information is extracted. For global interactions, a lane convolutional network is combined with a multi-head attention layer. The lane convolutional network captures local interactions between the agent and its surroundings, while a self-attention mechanism is used to model global interactions between agents, further enhancing the model's predictive capabilities. A two-stage decoding process is designed in the decoder, dividing the decoding process into an initial target candidate generation phase and a refined classification and regression phase. This approach results in predicted trajectories with higher accuracy and greater robustness. This innovative graph structure, information interaction mechanism, and unique two-stage decoder architecture enable more accurate and robust prediction of the agent's future trajectory, particularly in complex intersection scenarios.

[0257] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A graph-centric vehicle multimodal trajectory prediction method, characterized by: include: Obtaining historical trajectories of vehicles participating in the traffic and lane lines corresponding to the historical trajectories, and dividing the lane lines corresponding to the historical trajectories into lane segments; Using the center point of the lane segment as a node and the positional relationship between the center points of adjacent lane segments as an edge, a local graph of each participating vehicle is constructed; After combining the geometric features and type features of lane segments and the dynamic interaction relationship between lane segments and participating vehicles, feature extraction is performed to obtain node features of the local graph. Construct a global lane map containing all lanes in the scene, encode the local map of each participating vehicle separately, and project all the encoded local maps onto the global lane map to obtain a global map; After aggregating the node features of all local graphs, lane convolution is performed on the aggregated feature vector to obtain a local feature fusion vector; Performing feature extraction on the local feature fusion vector through a global interaction relationship encoding layer to obtain a global interaction feature vector; Weightedly combining the local feature fusion vector and the global interaction feature vector to obtain node features of the global graph; Projecting the node features of the global graph back to the local graph of each traffic participating vehicle to obtain an enhanced local graph; The predicted trajectory of each traffic participant vehicle and the confidence level of each predicted trajectory are obtained according to the enhanced local graph.

2. The graph-centric vehicle multimodal trajectory prediction method according to claim 1, characterized in that: The method for encoding the local graph of each traffic participant vehicle is: A lane convolution operation is performed on the node features of the local graph, and an average pooling operation is performed after each layer of convolution, and the results of the convolution operation and the average pooling operation are added together to obtain an encoded local graph node feature vector.

3. The graph-centric vehicle multimodal trajectory prediction method according to claim 2, characterized in that: The global interaction relationship encoding layer is composed of a multi-head attention layer.

4. The graph-centric vehicle multimodal trajectory prediction method according to claim 3, characterized in that: The predicted trajectory of each participating vehicle and the confidence level of each predicted trajectory are obtained based on the enhanced local graph, including the following steps: Step 1: Use the first prediction unit to predict each node in the enhanced local graph, determine the probability of the node being the target point and the position residual from the node to the true target point; Step 2: Select the K nodes with the highest probability of being the target point as candidate target points, and pass the candidate target points through the multi-head attention layer to perform information interaction to obtain interaction features; Inputting the interactive features into a second prediction unit to obtain a probability of the candidate target point being the target point and a position residual from the candidate target point to the true target point; Step 3: Use the Bezier curve to fit the trajectory from the current coordinate point to the candidate target point to obtain the reference trajectory; Step 4: Fine-tune the reference trajectory to obtain a predicted trajectory, and use the probability of the candidate target point being the target point as the confidence of the corresponding predicted trajectory.

5. The graph-centric vehicle multimodal trajectory prediction method according to claim 4, characterized in that: The first prediction unit and the second prediction unit both adopt a multi-layer perceptron.

6. The graph-centric vehicle multimodal trajectory prediction method according to claim 4 or 5, characterized in that: In step 4, the reference trajectory is fine-tuned to obtain the predicted trajectory as follows: The vehicle's position at multiple future moments is obtained through the reference trajectory, the offset at each moment is determined through a multi-layer perceptron, and the position at multiple future moments is fine-tuned according to the offset to obtain a predicted trajectory.

Images