Multi-type agent trajectory prediction method and system fused with multi-graph convolutional network

Abstract

The invention discloses a multi-type agent trajectory prediction method and system fused with a multi-graph convolutional network, and the method comprises the following steps: S1, obtaining scene information and trajectory information of multiple types of traffic participants, and generating a data set; s2, generating scene feature maps according to scene information in the data set, and weighting and fusing the scene feature maps through the attention score to generate global scene information; s3, respectively extracting position information, speed information and direction information of various types of traffic participants in the data set, and constructing a multi-influence graph; s4, jointly embedding the processed multi-influence graph and global scene information into a space-time graph convolutional neural network, and performing coding processing through the space-time graph convolutional neural network to generate dynamic space-time features; and S5, decoding the dynamic spatio-temporal characteristics through the spatio-temporal transformation trajectory generation network to obtain future moment trajectory points of the traffic participants. According to the method, the dynamic interaction relationship among the multiple types of traffic participants can be more accurately captured in trajectory prediction of the multiple types of traffic participants, and the prediction precision and the real-time response capability in a complex traffic environment are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a method and system for predicting the trajectory of multiple types of intelligent agents by integrating multi-graph convolutional networks. Background Technology

[0002] Currently, methods for predicting the trajectories of multiple traffic participants can be broadly categorized into model-based methods and data-driven methods. Model-based methods typically rely on physical and dynamic models to describe the motion patterns of multiple traffic participants, considering factors such as their speed and acceleration. These methods can predict trajectories in simple scenarios, but in complex traffic environments, the behavior of multiple traffic participants is not only affected by their own motion states but also strongly interfered with by the behavior of other traffic participants and environmental factors, resulting in poor prediction performance of traditional methods in these situations.

[0003] Data-driven methods for predicting the trajectories of multiple traffic participants extract motion features from training datasets and employ deep learning models to predict future trajectories. These methods are typically based on graph neural networks (GNNs) and recurrent neural networks (RNNs), and can capture the spatial and temporal relationships between different types of traffic participants to some extent. However, most data-driven methods still lack in-depth modeling of the complex environmental influences behind multiple types of traffic participants, especially regarding the interactions between different types of traffic participants and other traffic participants.

[0004] In real-world traffic scenarios, the behavior of various types of traffic participants is influenced not only by their own motion states but also by the behavior of other traffic participants and the surrounding environment. However, most current methods still focus on modeling single spatial relationships, failing to effectively consider the complex environment and semantic information of the scene behind the behavior of various types of traffic participants. There is an urgent need for a method that can comprehensively consider the complex environment among various types of traffic participants and combine scene context information to improve the accuracy, interpretability, and real-time response capabilities of predictions. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for predicting the trajectory of multiple types of intelligent agents by integrating multi-graph convolutional networks. By combining scene attention mechanism and spatiotemporal graph convolutional network, the trajectory prediction of multiple types of traffic participants can more accurately capture the dynamic interaction relationship between multiple types of traffic participants, improve the prediction accuracy and real-time response capability in complex traffic environments, and thus improve the safety and decision-making capability of autonomous vehicles.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for predicting the trajectory of multiple types of intelligent agents by incorporating multi-graph convolutional networks includes the following steps:

[0008] S1. Obtain scene information and trajectory information of multiple types of traffic participants to generate a dataset;

[0009] S2. Generate scene feature maps based on scene information in the dataset. Generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants. Weight and fuse scene feature maps using attention scores to generate global scene information, which is then used as node features of the spatiotemporal graph convolutional neural network.

[0010] S3. Extract the location, speed, and direction information of each type of traffic participant from the dataset, construct the corresponding location influence map (PIG), speed influence map (VIG), and direction influence map (DIG), and fuse and normalize the above multi-influence maps.

[0011] S4. The processed multi-influence map and global scene information are jointly embedded into the spatiotemporal graph convolutional neural network, and the spatiotemporal graph convolutional neural network is used for encoding to generate dynamic spatiotemporal features.

[0012] S5. The dynamic spatiotemporal features are converted into a sequence format by a spatiotemporal transformation trajectory generation network, and then decoded to obtain the future trajectory points of traffic participants, thereby realizing trajectory prediction of traffic participants.

[0013] As a further aspect of the present invention, the specific steps of step S2 are as follows:

[0014] The semantic features of scene images are extracted from the dataset using a VGG convolutional neural network. After convolution and pooling operations, a scene feature map V is generated. fea Its expression is:

[0015] V fea =VGG-Net(I scene W vgg ),

[0016] Among them, V fea It is a scene feature map extracted using VGG-Net, I scene It is the input scene image, W vgg These are the pre-trained weights of VGG-Net;

[0017] The trajectory coordinates p of traffic participants are obtained through a multilayer perceptron (MLP). i Mapping to a high-dimensional space to form a location embedding vector, and then mapping the location embedding vector to the scene feature map V feaMatrix multiplication is performed to generate the raw attention score. The weight distribution is then adjusted using a multilayer perceptron (MLP), and after softmax normalization, the attention score C is obtained. i Its expression is:

[0018]

[0019] in, V represents the position embedding vector. fea W represents the scene feature map. att The weights representing the attention mechanism;

[0020] The calculated attention score C i With scene feature map V fea We perform weighting to obtain the weighted feature map V. weighted Its expression is:

[0021] V weighted =C i ·V fea ;

[0022] The weighted feature map V weighted Compared with the original scene feature map V fea Add them together to generate global scene information V. final .

[0023] As a further aspect of the present invention, the specific steps of step S3 are as follows:

[0024] S31: Construct a location influence map. By capturing the location coordinates of multiple types of traffic participants at different time steps, calculate the relative position between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0025]

[0026] in, This represents the Euclidean distance between traffic participants i and j.

[0027] Construct an initial relative position adjacency matrix based on the relative positions. The expression representing the positional interaction relationships between traffic participants is:

[0028]

[0029] S32: Construct a velocity impact map. By capturing the velocity vectors of traffic participants at different time steps, calculate the relative velocity between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0030]

[0031] Construct an initial relative velocity adjacency matrix based on the relative velocities. The expression representing the speed interaction relationship among multiple types of traffic participants is as follows:

[0032]

[0033] S33: Construct a direction influence map. By capturing the direction vectors of traffic participants at different time steps, calculate the relative direction between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0034]

[0035] Construct an initial relative direction adjacency matrix based on the relative directions. The expression representing the directional interaction relationships between traffic participants is:

[0036]

[0037] S34: Connect the initial relative position adjacency matrix Initial relative velocity adjacency matrix and the initial relative direction adjacency matrix Stacked into a multi-relation matrix The matrix covers full-dimensional interaction of position, velocity, and direction, and its expression is:

[0038]

[0039] Through the degree matrix Q t Multiple relation matrices Normalization is performed, and its expression is:

[0040]

[0041] Where I is the identity matrix.

[0042] As a further aspect of the present invention, the spatiotemporal graph convolutional neural network in step S4 includes a spatial graph convolutional network (S-GCN) module and a temporal graph convolutional network (T-GCN) module. The spatial graph convolutional network (S-GCN) module is used to process the multiple relation matrices in step S3. Each adjacency matrix is ​​smoothed with Gaussian kernel, self-connected, and residual connected;

[0043] Then, a spatial graph convolution operation is performed independently on each adjacency matrix to generate node features for the three types of influence graphs. The expression for the spatial graph convolution operation is as follows:

[0044]

[0045] in, This represents the node features output by the l-th layer. The output node features of the (l+1)th layer are represented by σ, where σ represents the corresponding activation function. This represents summing the results over the neighboring nodes u of node v. W represents the normalization coefficient. (l) This represents the weight matrix of the l-th layer;

[0046] The node features of the three types of influence maps are dynamically fused, and the size of the feature fusion map is adjusted by convolution, and then combined with the global scene information V. final Overlaying and fusion are used to extract global node spatial features;

[0047] The extracted global node spatial features are used as input to the graph convolutional network (GCN) in the temporal graph convolutional network (T-GCN) module to perform graph convolution operations, aggregate node neighborhood information, and obtain the final spatial features H. spatial The expression for the convolution operation of each graph layer is:

[0048]

[0049] in, Let W be the degree matrix, σ represent the corresponding activation function, and W (l) For learnable weights, It is an adjacency matrix with self-loops;

[0050] The final spatial features are reshaped into time series features H. time One-dimensional convolution operation is performed using a dilated convolution factor to generate the time series feature H after dilation and convolution. dilated The expression for the dilated convolution operation is:

[0051] H dilated =Conv1D(H time ,kernel_size=k,dilation=d),

[0052] Among them, H dilated This represents the time series features after dilation and convolution. Conv1D represents a one-dimensional convolution operation used to process one-dimensional time series data. H time The input time series features are usually represented by a matrix composed of the time step and the feature dimension. kernel_size = k represents the size of the kernel, which determines the width of the convolution window. dilation = d represents the dilation factor, which determines the step size of the convolution kernel on the time series. The larger the dilation factor, the wider the time step covered by the convolution kernel.

[0053] The time series features H after dilation and convolution are processed by the gated recursion (GRU) module in the Time-Graph Convolutional Network (T-GCN) module. dilated With historical hidden state h t-1 Combine, and update gate u t and reset door r t Control information transmission and generate the hidden state h at the current moment. t Its expression is:

[0054]

[0055] u t =σ(W u *[H dilated ,h t-1 ]),

[0056] r t =σ(W r *[H dilated ,h t-1 ]);

[0057] Residual blocks are introduced between stacked layers in the Temporal Graph Convolutional Network (T-GCN) module to ultimately generate dynamic spatiotemporal features H. final .

[0058] As a further aspect of the present invention, the spatiotemporal transformation trajectory generation network includes a spatiotemporal attention layer and a nonlinear trajectory generation module, which converts dynamic spatiotemporal features H... final As a comprehensive feature representation of traffic participants, it is input into the spatiotemporal attention layer to generate processed dynamic temporal features X. STA Its expression is:

[0059] X STA =Attention(X)⊙α(t),

[0060] Where α(t) is the time decay factor, X STA The dynamic temporal features after processing by the spatiotemporal attention mechanism;

[0061] Next, the processed dynamic temporal features X STA Input the nonlinear trajectory generation module to generate trajectory prediction results, the expression of which is:

[0062] Y NTG =tanh(W1X) STA +b1)+ε,

[0063] Where W1 is the transformation matrix, b1 is the bias term, ε is the introduced random noise, and Y... NTG The trajectory prediction result;

[0064] Finally, the output module converts the temporal features into future trajectory information, the expression of which is:

[0065] P future =W2Y NTG +b2,

[0066] Where W2 is the transformation matrix of the output module, b2 is the bias term of the output module, and P future This is the information for the generated future trajectory.

[0067] As a further aspect of the present invention, the expression for the negative log-likelihood loss function of the spatiotemporal transformation trajectory generation network is as follows:

[0068]

[0069] Where η represents the network's learning parameters, T obs T represents the observation period. pred Indicates the predicted time period. μ represents the coordinates of the trajectory points. x express The mean, μ y express The mean, σ x express Standard deviation, σ y express The standard deviation, ρ represents and The correlation coefficient between them;

[0070] By minimizing the negative log-likelihood loss function, the network parameters are adjusted so that the generated Gaussian distribution can accurately predict the true trajectory.

[0071] A multi-type intelligent agent trajectory prediction system integrating multi-graph convolutional networks includes: a scene information acquisition and preprocessing module, used to acquire scene information and trajectory information of multiple types of traffic participants, and generate a dataset;

[0072] The scene attention mechanism module is used to generate scene feature maps from scene information in the dataset, generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants, and weight and fuse scene feature maps through attention scores to generate global scene information.

[0073] A multi-graph convolutional neural network is used to extract the location, speed, and direction information of multiple types of traffic participants in the dataset, and construct corresponding position influence maps (PIG), speed influence maps (VIG), and direction influence maps (DIG). The above multi-influence maps are then fused and normalized.

[0074] Spatiotemporal graph convolutional neural networks are used to encode jointly embedded multi-influence graphs and global scene information to generate dynamic spatiotemporal features;

[0075] The spatiotemporal transformation trajectory generation network is used to convert dynamic spatiotemporal features into a sequence format, and then decode them to obtain the future time trajectory points of traffic participants.

[0076] The modules described above achieve trajectory prediction for traffic participants by executing the multi-type agent trajectory prediction method that integrates multi-graph convolutional networks.

[0077] A storage medium storing computer instructions, which, when executed by a processor, perform the multi-type agent trajectory prediction method that integrates a multi-graph convolutional network.

[0078] Compared with the prior art, the beneficial effects of the present invention are:

[0079] 1. The multi-type traffic participant trajectory prediction method proposed in this invention integrates scene features and trajectory information of multi-type traffic participants, and captures the interaction between multi-type traffic participants on the road by constructing a multi-influence map. It can effectively solve the inaccuracy problem caused by ignoring the behavior and environmental interaction of multi-type traffic participants in traditional methods, and provide more accurate data support for multi-type traffic participant trajectory prediction.

[0080] 2. The trajectory prediction method proposed in this invention adopts a scene attention mechanism, which can dynamically adjust the weight of the impact on surrounding vehicles according to the real-time traffic scene and capture key changes in the environment, thereby improving the response speed to changes in the surrounding environment. This mechanism has good adaptability in highly dynamic and complex traffic environments, ensuring that the vehicle makes safe and reasonable driving decisions.

[0081] 3. The trajectory prediction method proposed in this invention can jointly learn the spatiotemporal relationship between the behavior of participants such as vehicles and pedestrians and environmental factors through spatiotemporal graph convolutional neural networks, reducing the misjudgment caused by ignoring the environmental context in traditional methods. The prediction accuracy of this model can be improved in complex traffic conditions, especially during lane change, the prediction of the behavior of surrounding vehicles is more accurate, thereby effectively avoiding potential collision risks. Attached Figure Description

[0082] Figure 1 This is a flowchart illustrating a multi-type agent trajectory prediction method that integrates multi-graph convolutional networks;

[0083] Figure 2 This is a schematic diagram of a multi-type intelligent agent trajectory prediction system that integrates multi-graph convolutional networks. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] Please see Figures 1-2 This invention provides a technical solution: a method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks, comprising the following steps:

[0086] S1. Obtain scene information and trajectory information of multiple types of traffic participants to generate a dataset;

[0087] As a specific solution, the specific content of step S1 is as follows: collect raw data of various types of traffic participants and their surrounding environment through multiple sensors (such as cameras, lidar, etc.), including trajectory information such as the position, speed, and posture of the participants, as well as static and dynamic features such as the location of buildings, the location of obstacles, and the road layout in the scene;

[0088] First, the collected raw data undergoes data cleaning and alignment, noise removal, missing value imputation, coordinate alignment, and time interpolation, and the trajectory data is smoothed. Next, key features are extracted, including historical trajectory coordinates (X,Y) and timestamps. Subsequently, a VGG convolutional neural network can be used to extract semantic features in the scene. In addition, data augmentation techniques can be used to expand the dataset, improve the model's generalization ability, and convert the data into a format suitable for model input, providing high-quality input data for trajectory prediction of various types of traffic participants (such as intelligent vehicles, pedestrians, etc.).

[0089] S2. Generate scene feature maps based on scene information in the dataset. Generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants. Weight and fuse scene feature maps using attention scores to generate global scene information, which is then used as node features of the spatiotemporal graph convolutional neural network.

[0090] As a specific solution, the specific steps of step S2 are as follows:

[0091] Semantic features of scene images are extracted from the dataset using a pre-trained VGG convolutional neural network. After convolution and pooling operations, a scene feature map V is generated. fea Its expression is:

[0092] V fea =VGG-Net(I scene W vgg ),

[0093] Among them, V fea It is a scene feature map extracted using VGG-Net, I scene It is the input scene image, W vgg These are the pre-trained weights of VGG-Net;

[0094] The trajectory coordinates p of traffic participants are obtained through a multilayer perceptron (MLP). i Mapping to a high-dimensional space creates a location embedding vector, which enhances the correlation between location and scene features. This location embedding vector is then mapped to the scene feature map V. fea Matrix multiplication is performed to generate the raw attention score. The weight distribution is then adjusted using a multilayer perceptron (MLP), and after softmax normalization, the attention score C is obtained. i This identifies key regions in the scene that are relevant to the current agent, and its expression is:

[0095]

[0096] in, V represents the position embedding vector. fea W represents the scene feature map. att The weights representing the attention mechanism;

[0097] The calculated attention score C i With scene feature map V fea We perform weighting to obtain the weighted feature map V. weighted Its expression is:

[0098] V weighted =C i ·V fea This operation highlights the most important areas for predicting the future trajectories of various traffic participants, while attenuating the feature values ​​of low-weight areas, thus emphasizing key targets and suppressing background information irrelevant to the prediction.

[0099] The weighted feature map V weighted Compared with the original scene feature map V fea Add them together to generate global scene information V. final This is a final multi-scale feature information containing important scene information. This feature information will be provided as input to the subsequent trajectory prediction model, and will be jointly modeled with the position / velocity / direction map to enhance the model's understanding of scene information and its ability to predict the behavior of multiple types of traffic participants.

[0100] S3. The location, speed and direction information of each type of traffic participant in the dataset are extracted by a multi-graph convolutional neural network, and the corresponding position influence map (PIG), speed influence map (VIG) and direction influence map (DIG) are constructed. The above multi-influence maps are fused and normalized. Each influence map is initialized by Euclidean distance, speed angle and relative speed difference. Each influence map can be updated in real time with time step to adapt to the changes in the motion state of traffic participants.

[0101] As a specific solution, the specific steps of step S3 are as follows:

[0102] S31. Construct a Location Influence Map (PIG). By capturing the location coordinates of various types of traffic participants at different time steps, calculate the relative position between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0103]

[0104] in, This represents the Euclidean distance between traffic participants i and j.

[0105] Construct an initial relative position adjacency matrix based on the relative positions. The expression representing the positional interaction relationships between traffic participants is:

[0106]

[0107] S32. Construct a Velocity Influence Map (VIG). By capturing the velocity vectors of traffic participants at different time steps, calculate the relative velocity between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0108]

[0109] Construct an initial relative velocity adjacency matrix based on the relative velocities. The expression representing the speed interaction relationship among multiple types of traffic participants is as follows:

[0110]

[0111] S33. Construct a Directional Influence Map (DIG). By capturing the direction vectors of traffic participants at different time steps, calculate the relative direction between traffic participant i and traffic participant j at each time step t. The expression is as follows:

[0112]

[0113] Construct an initial relative direction adjacency matrix based on the relative directions. The expression representing the directional interaction relationships between traffic participants is:

[0114]

[0115] S34. Convert the initial relative position adjacency matrix. Initial relative velocity adjacency matrix and the initial relative direction adjacency matrix Stacked into a multi-relation matrix The matrix covers full-dimensional interaction of position, velocity, and direction, and its expression is:

[0116]

[0117] Through the degree matrix Q t Multiple relation matrices Normalization is performed, and its expression is:

[0118] Where I is the identity matrix; normalization eliminates dimensional differences and improves model stability. This process not only balances the contribution weights of features, but also improves the stability and generalization ability of the model by suppressing the information propagation intensity of high-degree nodes and enhancing the signal of low-degree nodes, thus avoiding the problems of gradient explosion or vanishing.

[0119] S4. The processed multi-influence map and global scene information are jointly embedded into the spatiotemporal graph convolutional neural network, and the spatiotemporal graph convolutional neural network is used for encoding to generate dynamic spatiotemporal features.

[0120] As a specific implementation, the spatiotemporal graph convolutional neural network in step S4 includes a spatial graph convolutional network (S-GCN) module and a temporal graph convolutional network (T-GCN) module. The spatial graph convolutional network (S-GCN) module is used to process the multiple relation matrices in step S3. Each adjacency matrix undergoes Gaussian kernel smoothing, self-connection, and residual connection. The Gaussian kernel function performs element-wise multiplication on the initial adjacency matrix, calculates node feature similarity, and dynamically adjusts edge weights to adapt to different scene perception ranges. In the self-connection process, through… Introducing self-loops preserves the unique characteristics of nodes; residual connections can alleviate the gradient vanishing problem.

[0121] Then, spatial graph convolution operations are performed independently on each adjacency matrix to generate node features of the three types of influence graphs (e.g., node features). Node features Node features The expression for the spatial graph convolution operation is:

[0122]

[0123] in, This represents the node features output by the l-th layer. The output node features of the (l+1)th layer are represented by σ, where σ represents the corresponding activation function. This represents summing the results over the neighboring nodes u of node v. W represents the normalization coefficient. (l) This represents the weight matrix of the l-th layer;

[0124] In practical applications, to improve computational efficiency, each node only retains connections to its five nearest neighbor nodes, reducing computational complexity; at the same time, KD-Tree nearest neighbor search is used to quickly filter key neighbors, avoiding the computational complexity of a fully connected adjacency matrix.

[0125] Node features of the three types of influence maps (node ​​features) Node features Node features Dynamic fusion is performed to form a feature fusion map. Convolutional adjustment of feature fusion map The size of it is compared with the global scene information V. final Overlaying and fusion are used to extract global node spatial features;

[0126] The extracted global node spatial features are used as input to the graph convolutional network (GCN) in the temporal graph convolutional network (T-GCN) module to perform graph convolution operations, aggregate node neighborhood information, and obtain the final spatial features H. spatial The expression for the convolution operation of each graph layer is:

[0127]

[0128] in, Let W be the degree matrix, σ represent the corresponding activation function, and W (l) For learnable weights, It is an adjacency matrix with self-loops;

[0129] The final spatial feature H spatial Remodeling into time series features H time One-dimensional convolution operation is performed using a dilated convolution factor to generate the time series feature H after dilation and convolution. dilated The expression for the dilated convolution operation is:

[0130] H dilated =Conv1D(H time ,kernel_size=k,dilation=d),

[0131] Among them, H dilatedThis represents the time series features after dilation and convolution. Conv1D represents a one-dimensional convolution operation used to process one-dimensional time series data. H time The input time series features are typically represented by a matrix consisting of the time step and the feature dimension. kernel_size = k represents the kernel size, which determines the width of the convolution window. dilation = d represents the dilation factor, which determines the step size that the convolution kernel jumps across the time series. The larger the dilation factor, the wider the time step covered by the convolution kernel. Using the dilation factor for dilation convolution ensures that the output at the current moment depends only on historical information, avoiding future data leakage. The receptive field is expanded exponentially layer by layer to ensure that the output depends only on historical data.

[0132] The time series features H after dilation and convolution are processed by the gated recursion (GRU) module in the Time-Graph Convolutional Network (T-GCN) module. dilated With historical hidden state h t-1 Combine, and update gate u t and reset door r t Control information transmission and generate the hidden state h at the current moment. t Its expression is:

[0133]

[0134] u t =σ(W u *[H dilated ,h t-1 ]),

[0135] r t =σ(W r *[H dilated ,h t-1 ]);

[0136] Residual blocks are introduced between stacked layers in the Temporal Graph Convolutional Network (T-GCN) module to ultimately generate dynamic spatiotemporal features H. final .

[0137] S5. The dynamic spatiotemporal features are converted into a sequence format by a spatiotemporal transformation trajectory generation network, and then decoded to obtain the future trajectory points of traffic participants, thereby realizing trajectory prediction of traffic participants.

[0138] As a specific solution, the spatiotemporal transformation trajectory generation network includes a spatiotemporal attention layer and a nonlinear trajectory generation module, which incorporates dynamic spatiotemporal features H. final As a comprehensive feature representation of traffic participants, it is input into the spatiotemporal attention layer to generate processed dynamic temporal features X. STA Its expression is:

[0139] X STA=Attention(X)⊙α(t),

[0140] Where α(t) is the time decay factor, X STA The dynamic temporal features after processing by the spatiotemporal attention mechanism;

[0141] Next, the processed dynamic temporal features X STA Input the nonlinear trajectory generation module to generate trajectory prediction results, the expression of which is:

[0142] Y NTG =tanh(W1X) STA +b1)+ε,

[0143] Where W1 is the transformation matrix, b1 is the bias term, ε is the introduced random noise, and Y... NTG The trajectory prediction result;

[0144] Finally, the output module converts the temporal features into future trajectory information, the expression of which is:

[0145] P future =W2Y NTG +b2,

[0146] Where W2 is the transformation matrix of the output module, b2 is the bias term of the output module, and P future This is the information for the generated future trajectory.

[0147] As a further aspect of the present invention, the expression for the negative log-likelihood loss function of the spatiotemporal transformation trajectory generation network is as follows:

[0148]

[0149] Where η represents the network's learning parameters, T obs T represents the observation period. pred Indicates the predicted time period. μ represents the coordinates of the trajectory points. x express The mean, μ y express The mean, σ x express Standard deviation, σ y express The standard deviation, ρ represents and The correlation coefficient between them;

[0150] By minimizing the negative log-likelihood loss function, the network parameters are adjusted so that the generated Gaussian distribution can accurately predict the real trajectory. In practical applications, multiple future trajectories can be generated through sampling, and the most reasonable prediction result can be selected in the end.

[0151] This invention also proposes a multi-type intelligent agent trajectory prediction system that integrates multi-graph convolutional networks, including: a scene information acquisition and preprocessing module, used to acquire scene information and trajectory information of multiple types of traffic participants, and generate a dataset;

[0152] The scene attention mechanism module is used to generate scene feature maps from scene information in the dataset, generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants, and weight and fuse scene feature maps through attention scores to generate global scene information.

[0153] A multi-graph convolutional neural network is used to extract the location, speed, and direction information of multiple types of traffic participants in the dataset, and construct corresponding position influence maps (PIG), speed influence maps (VIG), and direction influence maps (DIG). The above multi-influence maps are then fused and normalized.

[0154] Spatiotemporal graph convolutional neural networks are used to encode jointly embedded multi-influence graphs and global scene information to generate dynamic spatiotemporal features;

[0155] The spatiotemporal transformation trajectory generation network is used to convert dynamic spatiotemporal features into a sequence format, and then decode them to obtain the future time trajectory points of traffic participants.

[0156] The modules described above achieve trajectory prediction for traffic participants by executing the multi-type agent trajectory prediction method that integrates multi-graph convolutional networks.

[0157] The present invention also proposes a storage medium storing computer instructions, which, when executed by a processor, execute the multi-type agent trajectory prediction method that integrates multi-graph convolutional networks.

[0158] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks, characterized in that, Includes the following steps: S1. Obtain scene information and trajectory information of multiple types of traffic participants to generate a dataset; S2. Generate scene feature maps based on scene information in the dataset. Generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants. Weight and fuse scene feature maps using attention scores to generate global scene information, which is then used as node features of the spatiotemporal graph convolutional neural network. S3. Extract the location, speed and direction information of each type of traffic participant from the dataset, construct the corresponding location influence map, speed influence map and direction influence map, and fuse and normalize the above multi-influence maps. S4. The processed multi-influence map and global scene information are jointly embedded into the spatiotemporal graph convolutional neural network, and the spatiotemporal graph convolutional neural network is used for encoding to generate dynamic spatiotemporal features. S5. The dynamic spatiotemporal features are converted into a sequence format by a spatiotemporal transformation trajectory generation network, and then decoded to obtain the future trajectory points of traffic participants, thereby realizing trajectory prediction of traffic participants.

2. The method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks according to claim 1, characterized in that, The specific steps of step S2 are as follows: The semantic features of scene images are extracted from the dataset using a VGG convolutional neural network. After convolution and pooling operations, a scene feature map V is generated. fea Its expression is: V fea =VGG-Net(I scene ;W vgg ), Among them, V fea It is a scene feature map extracted using VGG-Net, I scene It is the input scene image, W vgg These are the pre-trained weights of VGG-Net; The trajectory coordinates p of traffic participants are obtained through a multilayer perceptron. i Mapping to a high-dimensional space to form a location embedding vector, and then mapping the location embedding vector to the scene feature map V fea Matrix multiplication is performed to generate the raw attention score. The weight distribution is then adjusted using a multilayer perceptron, and after Softmax normalization, the attention score C is obtained. i Its expression is: in, V represents the position embedding vector. fea W represents the scene feature map. att The weights representing the attention mechanism; The calculated attention score C i With scene feature map V fea We perform weighting to obtain the weighted feature map V. weighted Its expression is: In weighted =C i ·In fea ; The weighted feature map V weighted Compared with the original scene feature map V fea Add them together to generate global scene information V. final .

3. The method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks according to claim 2, characterized in that, The specific steps of step S3 are as follows: S31: Construct a location influence map. By capturing the location coordinates of multiple types of traffic participants at different time steps, calculate the relative position between traffic participant i and traffic participant j at each time step t. The expression is as follows: in, This represents the Euclidean distance between traffic participants i and j. Construct an initial relative position adjacency matrix based on the relative positions. The expression representing the positional interaction relationships between traffic participants is: S32: Construct a velocity impact map. By capturing the velocity vectors of traffic participants at different time steps, calculate the relative velocity between traffic participant i and traffic participant j at each time step t. The expression is as follows: Construct an initial relative velocity adjacency matrix based on the relative velocities. The expression representing the speed interaction relationship among multiple types of traffic participants is as follows: S33: Construct a direction influence map. By capturing the direction vectors of traffic participants at different time steps, calculate the relative direction between traffic participant i and traffic participant j at each time step t. The expression is as follows: Construct an initial relative direction adjacency matrix based on the relative directions. The expression representing the directional interaction relationships between traffic participants is: S34: Connect the initial relative position adjacency matrix Initial relative velocity adjacency matrix and the initial relative direction adjacency matrix Stacked into a multi-relation matrix The matrix covers full-dimensional interaction of position, velocity, and direction, and its expression is: Through the degree matrix Q t Multiple relation matrices Normalization is performed, and its expression is: Where I is the identity matrix.

4. The method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks according to claim 3, characterized in that, The spatiotemporal graph convolutional neural network in step S4 includes a spatial graph convolutional network module and a temporal graph convolutional network module. The spatial graph convolutional network module is used to process the multiple relation matrices in step S3. Each adjacency matrix is ​​smoothed with Gaussian kernel, self-connected, and residual connected; Then, a spatial graph convolution operation is performed independently on each adjacency matrix to generate node features for the three types of influence graphs. The expression for the spatial graph convolution operation is as follows: in, This represents the node features output by the l-th layer. Let σ represent the node features output from layer (l+1), and let σ represent the corresponding activation function. This represents summing the results over the neighboring nodes u of node v. W represents the normalization coefficient. (l) This represents the weight matrix of the l-th layer; The node features of the three types of influence maps are dynamically fused, and the size of the feature fusion map is adjusted by convolution, and then combined with the global scene information V. final Overlaying and fusion are used to extract global node spatial features; The extracted global node spatial features are used as input to the graph convolutional network in the temporal graph convolutional network module to perform graph convolution operations, aggregate node neighborhood information, and obtain the final spatial features H. spatial The expression for the convolution operation of each graph layer is: in, Let W be the degree matrix, σ represent the corresponding activation function, and W (l) For learnable weights, It is an adjacency matrix with self-loops; The final spatial features are reshaped into time series features H. time One-dimensional convolution operation is performed using a dilated convolution factor to generate the time series feature H after dilation and convolution. dilated The expression for the dilated convolution operation is: H dilated =Conv1D(H time ,kernel_size=k,dilation=d), Among them, H dilated This represents the time series features after dilation and convolution. Conv1D represents a one-dimensional convolution operation used to process one-dimensional time series data. H time The input time series features are usually represented by a matrix composed of the time step and the feature dimension. kernel_size = k represents the size of the kernel, which determines the width of the convolution window. dilation = d represents the dilation factor, which determines the step size of the convolution kernel on the time series. The larger the dilation factor, the wider the time step covered by the convolution kernel. The time-series features H after dilation and convolution are processed by the gating mechanism module in the time-graph convolutional network module. dilated With historical hidden state h t-1 Combine, and update gate u t and reset door r t Control information transmission and generate the hidden state h at the current moment. t Its expression is: you t =σ(W u *[H dilated ,h t-1 ]), r t =σ(W r *[H dilated ,h t-1 ]); Residual blocks are introduced between stacked layers in the temporal graph convolutional network module to ultimately generate dynamic spatiotemporal features H. final .

5. The method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks according to claim 4, characterized in that, The spatiotemporal transformation trajectory generation network includes a spatiotemporal attention layer and a nonlinear trajectory generation module, which will generate dynamic spatiotemporal features H. final As a comprehensive feature representation of traffic participants, it is input into the spatiotemporal attention layer to generate processed dynamic temporal features X. STA Its expression is: X STA =Attention(X)⊙α(t), Where α(t) is the time decay factor, X STA The dynamic temporal features after processing by the spatiotemporal attention mechanism; Next, the processed dynamic temporal features X STA Input the nonlinear trajectory generation module to generate trajectory prediction results, the expression of which is: AND NTG =tanh(W1X STA +b1)+ε, Where W1 is the transformation matrix, b1 is the bias term, ε is the introduced random noise, and Y... NTG The trajectory prediction result; Finally, the output module converts the temporal features into future trajectory information, the expression of which is: P future W2Y NTG +b2, Where W2 is the transformation matrix of the output module, b2 is the bias term of the output module, and P future This is the information for the generated future trajectory.

6. The method for predicting the trajectory of multiple types of intelligent agents by fusing multi-graph convolutional networks according to claim 5, characterized in that, The expression for the negative log-likelihood loss function of the spatiotemporal transform trajectory generation network is as follows: Where η represents the network's learning parameters, T obs T represents the observation period. pred Indicates the predicted time period. μ represents the coordinates of the trajectory points. x express The mean, μ y express The mean, σ x express Standard deviation, σ y express The standard deviation, ρ represents and The correlation coefficient between them; By minimizing the negative log-likelihood loss function, the network parameters are adjusted so that the generated Gaussian distribution can accurately predict the true trajectory.

7. A trajectory prediction system for multiple types of intelligent agents integrating multi-graph convolutional networks, characterized in that, include: The scene information acquisition and preprocessing module is used to acquire scene information and trajectory information of multiple types of traffic participants to generate a dataset; The scene attention mechanism module is used to generate scene feature maps from scene information in the dataset, generate attention scores based on scene feature maps and trajectory coordinates of various types of traffic participants, and weight and fuse scene feature maps through attention scores to generate global scene information. A multi-graph convolutional neural network is used to extract the location, speed, and direction information of multiple types of traffic participants in the dataset, construct corresponding location influence maps, speed influence maps, and direction influence maps, and then fuse and normalize the above multi-influence maps. Spatiotemporal graph convolutional neural networks are used to encode jointly embedded multi-influence graphs and global scene information to generate dynamic spatiotemporal features; The spatiotemporal transformation trajectory generation network is used to convert dynamic spatiotemporal features into a sequence format, and then decode them to obtain the future time trajectory points of traffic participants. Each of the above modules implements the multi-type agent trajectory prediction method of the fusion multi-graph convolutional network as described in any one of claims 1-6 to predict the trajectory of traffic participants.

8. A storage medium, characterized in that, It stores computer instructions, which, when executed by a processor, perform the multi-type agent trajectory prediction method as described in any one of claims 1-6, which integrates multi-graph convolutional networks.

Images