Multimodal spatio-temporal model for precise motion prediction based on visual fusion

Abstract

This invention discloses a multimodal spatiotemporal model for accurate motion prediction based on visual fusion, comprising: an input encoder module: encoding the trajectory and map separately, and integrating the features of the trajectory and map through feature fusion; a space-time interaction module: extracting the space-time interaction behavior between vehicle trajectories by employing a space-time cross-attention mechanism, and outputting the optimized result through a gating mechanism; and a trajectory decoder module: generating multimodal trajectories using a linear residual layer, receiving the optimized output from the space-time interaction module, and predicting multiple possible trajectories for the vehicle; thereby capturing the spatiotemporal interaction between vehicles and improving prediction accuracy by utilizing the geographical structure of roads.

Description

Technical Field

[0001] This invention relates to the field of vehicle trajectory prediction technology in autonomous driving, and in particular to a multimodal spatiotemporal model for accurate motion prediction based on visual fusion. Background Technology

[0002] With the popularization of artificial intelligence technology, fields such as autonomous driving, intelligent transportation, robot control, and intelligent logistics are booming. Accurately predicting the trajectory of moving objects is crucial for ensuring traffic safety and efficiency. To achieve this goal, various sensors are used to collect data from the environment. This multi-source visual information is typically fused and understood with point cloud information from LiDAR and time-series data from other sensors to further represent rich traffic information. In fields such as intelligent logistics, accurate trajectory prediction is essential for achieving goals such as intelligent path planning and optimizing logistics transportation operations. Similarly, in autonomous driving and robot control, predicting the trajectories of other vehicles, pedestrians, and obstacles is crucial for ensuring safe and efficient navigation and control decisions. Therefore, the development of trajectory prediction tasks has significant application value and social implications.

[0003] A vehicle's future trajectory is typically influenced by three main factors: road structure, the vehicle's historical trajectory, and the trajectories of other vehicles in the vicinity. However, visual data is not always reliable. In scenarios such as nighttime, rain, or heavy fog, the quality of images captured by cameras can degrade significantly; sudden changes in light intensity can temporarily blind cameras—these are currently unresolved issues in the field of vision.

[0004] In recent years, a great deal of research and development work has emerged in the field of trajectory prediction, both in academia and industry. For example, Houenou et al. (Vehicle trajectory prediction based on motion model and maneuver recognition, 2013, 4363–4369) used a model that considers the instantaneous path of the vehicle, road shape, and vehicle handling for short-term trajectory prediction; MT Abbas et al. (An adaptive approach to vehicle trajectory prediction using multimodel Kalman filter, 2020) used a multimodal Kalman filter to handle a wider range of possible trajectory scenarios; N. Deo et al. (Convolutional social pooling for vehicle trajectory prediction, 2018, 1468–1476) further improved the model's ability to capture the interdependencies of vehicle motion by incorporating convolutional social pooling layers into the LSTM model; P. Pecher et al. (Data-driven vehicle trajectory prediction, 2016, 13–22) improved the accuracy of trajectory prediction by increasing the complexity of Markov models and neural network models; J. Mercat et al. (Multi-head attention for multi-modal joint vehicle motion forecasting, (2020, 9638–9644) proposed a prediction model that combines an LSTM encoder-decoder structure with a multi-head attention mechanism; M. Liang et al. (Learning lane graph representations for motion forecasting, 2020, 541–556) proposed LaneGCN, which considers multi-level interactions between map nodes and propagates vehicle trajectory features in the traffic network. It uses an attention mechanism to fuse data between vehicle nodes and lane maps to capture complex topological features; T. Gilles et al. (Home: Heatmap output for future motion estimation, 2021, 500–507) proposed HOME, which encodes history and local context, decodes the output heatmap using metric standards, uses fully connected layers for trajectory sampling, and finally decodes the final position of the trajectory; and so on.

[0005] The aforementioned methods primarily focus on trajectory prediction using different decoding techniques, which have certain limitations in modeling spatiotemporal relationships and fail to fully utilize the potential spatiotemporal dependencies inherent in multimodal trajectory data. Furthermore, they neglect the constraints of road geographical structure on vehicle trajectories, resulting in poor prediction performance. Therefore, for intelligent transportation systems, designing methods to effectively utilize multi-source, multimodal data to accurately predict the future motion trajectory of vehicles has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0006] To address the problems existing in the prior art, the purpose of this invention is to provide a multimodal spatiotemporal model for accurate motion prediction based on visual fusion, so as to capture the spatiotemporal interaction between vehicles and improve prediction accuracy by utilizing the geographical structure of roads.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a multimodal spatiotemporal model for accurate motion prediction based on visual fusion, comprising:

[0008] Input encoder module: Encodes the trajectory and map separately, and integrates the features of the trajectory and map together through feature fusion;

[0009] The space-time interaction module extracts the space-time interaction behaviors between vehicle trajectories by employing a space-time cross-attention mechanism, and outputs the optimized results through a gating mechanism.

[0010] The trajectory decoder module generates multimodal trajectories using linear residual layers, receives optimized outputs from the space-time interaction module, and predicts multiple possible trajectories for the vehicle.

[0011] As a further improvement of the present invention, the input encoder module specifically includes the following components:

[0012] Track encoder: To embed the trajectory sequence features of the vehicles, a stacked LSTM network was used; the historical trajectory of each vehicle is represented as follows. Each of them These represent the position coordinates, vehicle heading, instantaneous speed, and timestamp at different time segments; the network parameters are shared across all embedded trajector nodes, thus yielding a feature vector for each agent node. :

[0013] (1);

[0014] High-precision map encoder: To capture the adjacency relationships between lanes, four directed graphs are constructed, represented as follows: ,in The center point of the lane is represented; the directed graph represents the topology of the lane nodes and is described by four types of edge connections: pre, suc, left neighbor, and right neighbor; Lane graph convolution (LaneGCN) is used to aggregate node information, and considering the uncertainty and low-order characteristics of left and right neighbors, the convolution operation is only performed once; in order to consider the continuity of the road on the pre and suc nodes, dilated convolution is introduced to expand the receptive field in the lane direction; the convolution operation is repeated k times to effectively capture long-range dependencies along the lane direction; Lane graph convolution (LaneGCN) is expressed as Equation (2):

[0015] (2);

[0016] in, Represent node characteristics; The adjacency matrix representing type i relations; Representation matrix k-th power; These are the parameters used in training; Indicates the order of dilated convolution; dilation only occurs between predecessors and successors;

[0017] Fusion Network: LaneGraph Convolutional Network (LaneGCN) uses an attention mechanism for feature fusion, propagating vehicle features through interactions with lanes, including vehicle-lane, lane-lane, lane-vehicle, and vehicle-vehicle interactions; ultimately achieving comprehensive feature fusion between vehicles; where LaneGraph Convolutional Network (LaneGCN) is used for lane-lane feature fusion, and the attention fusion mechanism is used for vehicle-lane, lane-vehicle, and vehicle-vehicle feature fusion; specifically, for vehicle node i, the features of its context lane node j are aggregated according to the following formula (3):

[0018] (3);

[0019] in, Represent the characteristics of the i-th node; These are trainable parameters; It is the Euclidean distance between node i and node j; only when it satisfies Nodes that meet the threshold will be considered; This represents the operations of normalization and nonlinear activation functions.

[0020] As a further improvement of the present invention, the space-time interaction module specifically includes the following components:

[0021] Spatial Attention Module: Employs a spatial attention mechanism that calculates attention weights between different vehicles. These weights are then used to adjust the importance of each vehicle's information. The interaction relationships between vehicles in the environment are represented by constructing a spatiotemporal graph, as shown below. ; It contains various entities, and each smart agent is represented as a node; This indicates whether there is an interaction between different intelligent agents, which is determined based on the Euclidean distance between the coordinates of the intelligent agents; if the distance between two intelligent agents is less than a certain threshold, it indicates that there is an interaction between them; the spatial attention mechanism is only executed on the interacting vehicles, without considering all vehicles; each subgraph assumes that there is an interaction between all vehicles in it, thus constructing a bidirectional fully connected graph. As time goes by, the position of the intelligent agents changes, causing the interaction relationship to change as well. This process generates a series of spatial graphs that show the potential temporal relationships; the spatial attention module uses a graph convolutional network to effectively model the spatial relationships between vehicles; the calculation method of the graph convolutional network is described by formula (4):

[0022] (4);

[0023] in, , These are the adjacency matrix and the identity matrix; yes The degree matrix; These are the parameters used for training; Indicates the number of layers in the GCN;

[0024] Temporal attention module: used to enhance contextual information, promote a comprehensive understanding of trajectory behavior, and ultimately improve prediction accuracy; by integrating temporal attention, the model dynamically pays attention to trajectory information from different time periods and makes predictions based on temporal importance; vehicle features obtained from the upper layer are dynamically encoded through a multi-head attention layer; the single-head attention layer is represented by formula (5):

[0025] (5);

[0026] in, It is a non-linear activation function. It is a vector The dimension;

[0027] Multi-head attention is achieved by connecting the outputs of multiple single-head attention layers, as expressed in formula (6):

[0028] (6);

[0029] in, , These are trainable parameters; according to the model architecture diagram, it combines spatiotemporal cross-attention to obtain two sets of spatiotemporal feature sequences, among which... These are represented as the first features based on spatial priority and time priority, respectively; the spatiotemporal interaction module takes these features as input and generates fused features as the final output;

[0030] Geographic road gating: By designing gating parameters and introducing the concept of macroscopic connectivity, the adjacency matrix of the road network is obtained, denoted as... , expressed as formula (7):

[0031] (7);

[0032] Elements in the adjacency matrix lane and lane The connection between nodes; to quantitatively measure the strength of the association between two nodes, a parameter matrix is ​​introduced. ,in ; Represents the number of lanes in the road network; each element in the matrix Represents from the lane Transition to lane The possibility or probability; expressed as formula (8):

[0033] (8);

[0034] The modified linear unit (ReLU) activation function is used, and the lane connectivity matrix A is used as a mask; expressed as formula (9):

[0035] (9);

[0036] in It is the parameter matrix of the gating mechanism, and the symbol · represents dot product multiplication.

[0037] As a further improvement to the present invention, the design of the gating parameters specifically includes:

[0038] Gated vectors ,in It is a mapping function. It is the output of the spatiotemporal attention module; definition The function is a sigmoid function, expressed as formula (10):

[0039] (10);

[0040] By using gate vectors To correct the output of the spatiotemporal attention module, we obtain the output of the space-time interaction module. ;in, This is the output of the spatiotemporal attention module, and the symbol · represents the dot product operation.

[0041] As a further improvement of the present invention, in the trajectory decoder module, the encoded spatiotemporal trajectory features are decoded into a multimodal predicted trajectory sequence by using formula (11); the network architecture includes multiple linear layers, normalization layers and nonlinear activation layers; each residual connection connects the input and intermediate output of the model;

[0042] (11).

[0043] As a further improvement of the present invention, the multimodal spatiotemporal model is an end-to-end model, which adopts a smooth L1 loss function and specifically includes the following steps:

[0044] The expression for the smooth L1 loss function (12) is as follows:

[0045] (12);

[0046] in, Represents the predicted value at a specific time. Compared with the true value Euclidean distance error between .

[0047] Therefore, the loss function of the entire model is defined as formula (13):

[0048] (13);

[0049] in, This represents each time point at each position in the trajectory sequence.

[0050] The beneficial effects of this invention are:

[0051] This invention provides a multimodal spatiotemporal model for accurate motion prediction based on visual fusion. It integrates and represents multimodal data from multiple visual data sources and other sensors, thereby exploring the comprehensive value of multimodal data. MSVFU technology is employed to improve the accuracy and reliability of trajectory prediction. The model is based on graph neural networks and a spatial-temporal cross-attention mechanism, combined with high-precision map information, to extract and analyze the inherent spatiotemporal features in the integrated visual information and semantic content data. It uses a road gating mechanism to restrict the output to the road network, and finally predicts possible future multimodal driving trajectories through a residual network. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the framework of the multimodal spatiotemporal model in an embodiment of the present invention;

[0053] Figure 2 This is a spatial interaction diagram of the multimodal spatiotemporal model in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the time attention module of the multimodal spatiotemporal model in an embodiment of the present invention. Detailed Implementation

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] Example

[0057] Reference Figures 1 to 3 A multimodal spatiotemporal model for accurate motion prediction based on visual fusion, which consists of three main components:

[0058] Input encoder module: The encoder encodes the trajectory and the map separately, and integrates their features together through feature fusion.

[0059] Spatial-temporal interaction module: The interaction module extracts the spatial-temporal interaction behaviors between vehicle trajectories by adopting a spatial-temporal cross-attention mechanism, and outputs the optimized results through a gating mechanism.

[0060] Trajectory Decoder Module: The decoder uses a linear residual layer to generate multimodal trajectories. It receives the optimized output from the interactive module and predicts multiple possible trajectories for the vehicle. The linear residual layer helps capture residual information and improves predictions.

[0061] As described above, the beneficial effects of this invention are as follows: by integrating and representing multimodal data from multiple visual data sources and other sensors, the comprehensive value of multimodal data is explored, and the accuracy and reliability of trajectory prediction are improved by employing MSVFU technology. This model is based on graph neural networks and a space-time cross-attention mechanism, combined with high-precision map information, to extract and analyze the inherent spatiotemporal features in the integrated visual information and semantic content data. It uses a road gating mechanism to restrict the output to the road network, and finally predicts possible future multimodal driving trajectories through a residual network. Extensive testing was conducted on the publicly available real-world dataset Argoverse 2, and the results show that the proposed model achieves good performance.

[0062] Furthermore, the component input encoder module encodes the trajectory and map separately and integrates their features, specifically including the following components:

[0063] Trajectory encoder. To embed the trajectory sequence features of the vehicles, a stacked LSTM network is used, where the historical trajectory of each vehicle is represented as... Each of them These represent the position coordinates, vehicle heading, instantaneous speed, and timestamp at different time segments. The network parameters are shared across all embedded trajector nodes, thus yielding a feature vector for each proxy node. .

[0064] (1);

[0065] Furthermore, the input encoder module also includes:

[0066] High-precision map encoder. High-precision maps contain a wealth of semantic information, including lane locations, lane adjacency relationships, and intersection markings. To capture the adjacency relationships between lanes, four directed graphs were constructed, represented as follows: ,in The center point of the lane is represented. These graphs represent the topology of lane nodes and are described by four types of edge connections: pre, suc, left neighbor, and right neighbor. The LaneGCN (Lane Graph Convolution) method is used to aggregate node information. Considering the uncertainty and low-order properties of left and right neighbors, the convolution operation is performed only once. To consider the continuity of the road on the pre and suc nodes, dilated convolution is introduced to expand the receptive field in the lane direction. The convolution operation is repeated k times to effectively capture long-range dependencies along the lane direction. LaneGCN can be expressed as Equation (2):

[0067] (2);

[0068] in, Represent node characteristics; The adjacency matrix representing type i relations; Representation matrix k-th power; These are the parameters used in training; This indicates the order of the dilated convolution. Dilation only occurs between predecessors and successors.

[0069] Furthermore, the input encoder module also includes:

[0070] Fusion Network. Based on LaneGCN, a more general attention mechanism is adopted for feature fusion. This feature fusion aims to propagate vehicle features through interactions with lanes, including vehicle-lane, lane-lane, lane-vehicle, and vehicle-vehicle interactions. The ultimate goal is to achieve comprehensive feature fusion between vehicles. Specifically, LaneGCN is used for lane-lane feature fusion, while the attention fusion mechanism is used for vehicle-lane, lane-vehicle, and vehicle-vehicle feature fusion. For example, for vehicle node i, the features of its context lane node j can be aggregated according to the following formula (3):

[0071] (3);

[0072] in, This represents the feature of the i-th node. These are trainable parameters. It is the Euclidean distance between node i and node j. Only when it satisfies Threshold nodes will be considered, with thresholds set to 7 meters, 6 meters, and 100 meters for vehicle-lane, lane-vehicle, and vehicle-vehicle interactions, respectively. This represents the operations of normalization and nonlinear activation functions.

[0073] During propagation, lane and vehicle information are fully integrated and transmitted. This integration allows features from both sources to be effectively combined, resulting in a more comprehensive understanding of the interaction between vehicles and lanes. By fusing vehicle and lane information, the model can better grasp the complex relationship between vehicles and their surroundings, improving the performance of trajectory prediction models.

[0074] Furthermore, the spatial-temporal interaction module employs a spatial-temporal cross-attention mechanism to extract the spatiotemporal interaction behaviors between vehicle trajectories, and outputs the optimized results through a gating mechanism. Specifically, it includes the following components:

[0075] Spatial Attention Module. In trajectory prediction tasks, considering the diverse interactions between vehicles and other objects is crucial. To effectively capture and extract spatial dependencies between adjacent vehicles, a spatial attention mechanism is employed. This mechanism calculates attention weights between different vehicles. These weights are then used to adjust the importance of information about each vehicle, better reflecting the relationships between them. By introducing this mechanism, the model can more accurately predict the trajectory of each vehicle, thereby improving overall prediction performance. Through the spatial attention mechanism, the model can model spatial relationships in traffic scenes, thus better understanding the interactive behaviors between vehicles and positively impacting trajectory prediction tasks. The interactive relationships between vehicles in the environment can be effectively represented by constructing a spatiotemporal graph, as shown below. . It includes various entities, such as vehicles and pedestrians, and each smart agent is represented as a node. This indicates whether there is an interaction between different intelligent agents, which is determined based on the Euclidean distance between the coordinates of the intelligent agents. If the distance between two intelligent agents is less than a certain threshold, it indicates that there is an interaction between them, which may be an interaction between vehicles or an interaction between a vehicle and a pedestrian. The spatial attention mechanism is only executed on the interacting vehicles, without considering all vehicles. This method reduces the number of parameters and enhances computational efficiency. Each subgraph assumes that there is an interaction between all vehicles in it, thus constructing a bidirectional fully connected graph. As time goes by, the positions of the intelligent agents change, causing their interaction relationships to change as well. This process generates a series of spatial graphs that show the potential temporal relationships. The spatial attention module uses a graph convolutional network to effectively model the spatial relationships between vehicles. The computation method of the graph convolutional network can be described by formula (4):

[0076] (4);

[0077] in , It consists of the adjacency matrix and the identity matrix. yes The degree matrix. These are the parameters used for training. The spatial attention module stacks multiple layers of GCN to explore multi-level interactions between vehicles. The parameter represents the number of layers in the GCN, and we set it to 2. The non-linear activation function ReLU is used to connect each layer of the GCN.

[0078] Furthermore, the space-time interaction module also includes:

[0079] Temporal Attention Module. In the context of trajectory prediction for autonomous vehicles, existing methods mainly focus on the spatial structure of lanes and the spatial relationships between vehicles. However, they often lack comprehensive modeling of time-series features. When vehicle speeds are similar, relative spatial positions tend to change little, resulting in graph structures that are approximately or even identical. However, different time periods in the trajectory may have different features. For example, peak and off-peak traffic flow and driving speeds may be significantly different, leading to obvious differences in the features observed in the trajectory. To alleviate this problem, we propose introducing a temporal attention module, which aims to enhance contextual information, promote a comprehensive understanding of trajectory behavior, and ultimately improve prediction accuracy. By integrating temporal attention, the model can dynamically focus on trajectory information from different time periods and make predictions based on their temporal importance. Vehicle features obtained from the upper layer will be dynamically encoded through a multi-head attention layer. This method enhances the effectiveness and accuracy of the model while mitigating the risk of gradient vanishing or gradient exploding. The single-head attention layer can be expressed as Equation (5):

[0080] (5);

[0081] in It is a non-linear activation function. It is a vector The dimensions. Query, key, and value vectors can be calculated from the input sequence vector.

[0082] Multi-head attention is achieved by connecting the outputs of multiple single-head attention layers. This can be expressed as formula (6):

[0083] (6);

[0084] in , These are trainable parameters. According to the model architecture diagram, it combines spatiotemporal cross-attention to obtain two sets of spatiotemporal feature sequences, where... These are represented as the first features based on spatial priority and time priority, respectively. The spatiotemporal interaction module takes these features as input and generates a fused feature as the final output.

[0085] Furthermore, the space-time interaction module also includes:

[0086] Geographic road gating. Design gating parameters: By introducing the concept of macroscopic connectivity, we can obtain the adjacency matrix of the road network, denoted as... It can be expressed as formula (7):

[0087] (7);

[0088] Elements in the adjacency matrix lane and lane The connections between nodes are not limited to specific relationships such as successor, predecessor, left, right lane, etc. To quantitatively measure the strength of the association between two nodes, a parameter matrix is ​​introduced. ,in . here, This represents the number of lanes in the road network. Each element in the matrix... Represents from the lane Transition to lane The possibility or probability. It can be expressed as formula (8):

[0089] (8);

[0090] We employed the Modified Linear Unit (ReLU) activation function and used the lane connectivity matrix A as a mask. This means that during the computation, only the adjacency matrix is ​​considered. The feature matrix elements corresponding to the lanes connected in the middle. This helps to limit the model's attention to the relevant lanes, improving the efficiency of information dissemination, and can be expressed as formula (9):

[0091] (9);

[0092] in It is the parameter matrix of the gating mechanism, and the symbol · represents dot product multiplication.

[0093] Furthermore, the design of the gating function includes: gating vector ,in It is a mapping function. This is the output of the spatiotemporal attention module. Because... This directly reflects the strength of the connection between the two lanes; a larger value indicates a stronger connection. To ensure... The output is in the range [0,1], we define The function is a sigmoid function, which can be expressed as formula (10):

[0094] (10);

[0095] Therefore, by using a gate vector By correcting the output of the spatiotemporal attention module, we obtained the output of the interaction module. .in, This is the output of the spatiotemporal attention module, and the symbol · represents the dot product operation.

[0096] Furthermore, the trajectory decoder module utilizes a linear residual layer to generate a multimodal trajectory, receives optimized output from the interaction module, and predicts multiple possible trajectories for the vehicle. Specifically, it includes the following components:

[0097] In real-world scenarios, vehicles exhibit diverse driving intentions. For example, at an intersection, a vehicle can choose to go straight, turn left, turn right, or make a U-turn. These choices lead to different trajectories and multiple potential destinations. Furthermore, vehicles heading towards the same destination may choose different routes and change lanes at different locations, resulting in different trajectory sequences. To address the challenge of multimodal trajectory sequence prediction, the spatiotemporal features of each vehicle are simultaneously input into multiple residual linear layers. These layers produce multiple predictions to represent different possible outcomes. By using Equation (11), the encoded spatiotemporal trajectory features are decoded to generate a multimodal predicted trajectory sequence. The network architecture includes multiple linear layers, normalization layers, and nonlinear activation layers. Each residual connection connects the model's input and intermediate output.

[0098] (11);

[0099] Using residual structures in neural networks helps in learning nonlinear transformations across multiple layers. This structure helps mitigate the vanishing and exploding gradient problems, enhancing the expressive power of neural networks. Through residual connections, networks can more effectively learn and represent complex function mappings, thereby improving model performance and training effectiveness.

[0100] Furthermore, the end-to-end model proposed in this invention employs a smooth L1 loss function, which is a regression loss function, and specifically includes the following steps:

[0101] Compared with commonly used loss functions such as mean absolute error (MAE) and mean squared error (MSE), the smooth L1 loss function provides stronger robustness and computational efficiency for regression models. It can also better handle outliers in the dataset. The expression for the smooth L1 loss function (12) is as follows:

[0102] (12);

[0103] in, Represents the predicted value at a specific time. Compared with the true value Euclidean distance error between .

[0104] Therefore, the loss function of the entire model can be defined as formula (13):

[0105] (13);

[0106] in, This represents each time point at each position in the trajectory sequence.

[0107] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal spatiotemporal model for accurate motion prediction based on visual fusion, characterized in that, include: Input encoder module: Encodes the trajectory and map separately, and integrates the features of the trajectory and map together through feature fusion; The space-time interaction module extracts the space-time interaction behaviors between vehicle trajectories by employing a space-time cross-attention mechanism, and outputs the optimized results through a gating mechanism. The space-time interaction module specifically includes the following components: Spatial Attention Module: Employs a spatial attention mechanism that calculates attention weights between different vehicles. These weights are then used to adjust the importance of each vehicle's information. The interaction relationships between vehicles in the environment are represented by constructing a spatiotemporal graph, as shown below. ; It contains various entities, and each smart agent is represented as a node; This indicates whether there is an interaction between different intelligent agents, which is determined based on the Euclidean distance between the coordinates of the intelligent agents; if the distance between two intelligent agents is less than a certain threshold, it indicates that there is an interaction between them; the spatial attention mechanism is only executed on the interacting vehicles, without considering all vehicles; each subgraph assumes that there is an interaction between all vehicles in it, thus constructing a bidirectional fully connected graph. As time goes by, the position of the intelligent agents changes, causing the interaction relationship to change as well. This process generates a series of spatial graphs that show the potential temporal relationships; the spatial attention module uses a graph convolutional network to effectively model the spatial relationships between vehicles; the calculation method of the graph convolutional network is described by formula (4): （4）； in, , These are the adjacency matrix and the identity matrix; yes The degree matrix; These are the parameters used for training; Temporal attention module: used to enhance contextual information, promote a comprehensive understanding of trajectory behavior, and ultimately improve prediction accuracy; by integrating temporal attention, the model dynamically pays attention to trajectory information from different time periods and makes predictions based on temporal importance; vehicle features obtained from the upper layer are dynamically encoded through a multi-head attention layer; the single-head attention layer is represented by formula (5): （5）； in, It is a non-linear activation function. It is a vector The dimension; Multi-head attention is achieved by connecting the outputs of multiple single-head attention layers, as expressed in formula (6): （6）； in, , These are trainable parameters; the spatiotemporal interaction module combines spatiotemporal cross-attention to obtain two sets of spatiotemporal feature sequences, among which... These are represented as the first features based on spatial priority and time priority, respectively; the spatiotemporal interaction module takes these features as input and generates fused features as the final output; Geographic road gating: By designing gating parameters and introducing the concept of macroscopic connectivity, the adjacency matrix of the road network is obtained, denoted as... , expressed as formula (7): （7）； Elements in the adjacency matrix lane and lane The connection between nodes; to quantitatively measure the strength of the association between two nodes, a parameter matrix is ​​introduced. ,in ; Represents the number of lanes in the road network; each element in the matrix Represents from the lane Transition to lane The possibility or probability; expressed as formula (8): （8）； The modified linear unit (ReLU) activation function is used, and the lane connectivity matrix A is used as a mask; expressed as formula (9): （9）； in It is the parameter matrix of the gating mechanism, and the symbol · represents dot product multiplication; The design of gating parameters specifically includes: Gated vectors ,in It is a mapping function. It is the output of the spatiotemporal attention module; definition The function is a sigmoid function, expressed as formula (10): （10）； By using gate vectors To correct the output of the spatiotemporal attention module, we obtain the output of the space-time interaction module. ;in, This is the output of the spatiotemporal attention module; the symbol · represents the dot product operation. The trajectory decoder module generates multimodal trajectories using linear residual layers, receives optimized outputs from the space-time interaction module, and predicts multiple possible trajectories for the vehicle.

2. The multimodal spatiotemporal model for accurate motion prediction based on visual fusion according to claim 1, characterized in that, The input encoder module specifically includes the following components: Track encoder: To embed the trajectory sequence features of the vehicles, a stacked LSTM network was used; the historical trajectory of each vehicle is represented as follows. Each of them These represent the position coordinates, vehicle heading, instantaneous speed, and timestamp at different time segments; the network parameters are shared across all embedded trajector nodes, thus yielding a feature vector for each agent node. : （1）； High-precision map encoder: To capture the adjacency relationships between lanes, four directed graphs are constructed, represented as follows: ,in The center point of the lane is represented; the directed graph represents the topology of the lane nodes and is described by four types of edge connections: pre, suc, left neighbor, and right neighbor; Lane graph convolution (LaneGCN) is used to aggregate node information, and considering the uncertainty and low-order characteristics of left and right neighbors, the convolution operation is only performed once; in order to consider the continuity of the road on the pre and suc nodes, dilated convolution is introduced to expand the receptive field in the lane direction; the convolution operation is repeated k times to effectively capture long-range dependencies along the lane direction; Lane graph convolution (LaneGCN) is expressed as Equation (2): （2）； in, Represent node characteristics; The adjacency matrix representing type i relations; Representation matrix The power of k; W0 is a trainable parameter; Indicates the order of dilated convolution; dilation only occurs between predecessors and successors; Fusion Network: LaneGraph Convolutional Network (LaneGCN) uses an attention mechanism for feature fusion, propagating vehicle features through interactions with lanes, including vehicle-lane, lane-lane, lane-vehicle, and vehicle-vehicle interactions; ultimately achieving comprehensive feature fusion between vehicles; where LaneGraph Convolutional Network (LaneGCN) is used for lane-lane feature fusion, and the attention fusion mechanism is used for vehicle-lane, lane-vehicle, and vehicle-vehicle feature fusion; specifically, for vehicle node i, the features of its context lane node j are aggregated according to the following formula (3): （3）； in, Represent the characteristics of the i-th node; These are trainable parameters; It is the Euclidean distance between node i and node j; only when it satisfies Nodes that meet the threshold will be considered; This represents the operations of normalization and nonlinear activation functions.

3. The multimodal spatiotemporal model for accurate motion prediction based on visual fusion according to claim 1, characterized in that, In the trajectory decoder module, the encoded spatiotemporal trajectory features are decoded into a multimodal predicted trajectory sequence by using formula (11); the network architecture includes multiple linear layers, normalization layers and nonlinear activation layers; each residual connection connects the model's input and intermediate output. （11）。 4. The multimodal spatiotemporal model for accurate motion prediction based on visual fusion according to claim 1, characterized in that, The multimodal spatiotemporal model is an end-to-end model that employs a smooth L1 loss function, and specifically includes the following steps: The expression for the smooth L1 loss function (12) is as follows: （12）； in, Represents the predicted value at a specific time. Compared with the true value Euclidean distance error between ; Therefore, the loss function of the entire model is defined as formula (13): （13）； in, This represents each time point at each position in the trajectory sequence.

Images