Multi-agent motion prediction method based on dynamic graph and progressive multi-scale decoding

Abstract

The invention relates to a multi-agent joint motion prediction method based on a dynamic graph and progressive multi-scale decoding. The method comprises the following steps: acquiring agent features; the features of the intelligent agents are input into a decoder, the decoder comprises a time sequence module, a dynamic graph progressive construction module, a heterogeneous graph convolution module and prediction taps, the prediction taps comprise rough prediction taps and combined prediction taps, the decoder outputs the combined future state of the multiple intelligent agents, the traffic scene is changed, and the steps are repeated. The method comprises the following steps of: obtaining a joint future state of an intelligent agent in K different scenes, calculating a loss function based on the joint future state to train a scene encoder and a decoder, obtaining a motion prediction model, obtaining actual intelligent agent characteristics by the motion prediction model, and outputting an intelligent agent joint motion prediction result. Compared with the prior art, the multi-agent motion prediction method has the advantages of improving the continuity and accuracy of multi-agent motion prediction and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous vehicle motion prediction, and in particular to a multi-agent joint motion prediction method based on dynamic graphs and progressive multi-scale decoding. Background Art

[0002] Accurately predicting the motion of surrounding traffic participants is crucial for safe planning for autonomous vehicles. Recent research advances have expanded prediction techniques from the motion of a single traffic participant to the joint prediction of multiple interacting traffic participants, and various strategies have been proposed to handle the complex interactions among traffic participants' future motions. However, these approaches ignore the dynamic nature of these interactions.

[0003] Motion prediction is crucial for ensuring safe and efficient motion planning in autonomous driving systems. It predicts the future trajectories of surrounding traffic participants (such as vehicles and pedestrians, collectively referred to as agents) based on their historical trajectories and environmental information such as maps. Multi-agent joint motion prediction further simultaneously predicts the future motion of each traffic participant and their interactions. This requires both producing reasonable predictions for each traffic participant and ensuring the consistency of the predicted trajectories of different traffic participants, effectively avoiding collisions.

[0004] This task is challenging and far from being well solved, mainly due to the following characteristics of the task itself: (1) The uncertainty and multimodality of driving intentions and styles. For example, the driver may choose to go straight or turn at an intersection. (2) The dynamics and complexity of future interactions. The agent has various types of interactions with surrounding vehicles, pedestrians, and roads, and these interactions continue to evolve and change as the agent moves. For example, at an intersection, considering the dynamic interactions near the intersection is crucial for collision avoidance, such as vehicle A intending to turn left while vehicle B plans to go straight. (3) Scalability. As the number of agents increases, the joint trajectory space expands exponentially, making the interactions between them increasingly complex.

[0005] To address the above problems, some methods based on graph neural networks and graph transformers were first proposed. For example, HiVT[1] uses vector transformers to encode the historical motion trajectories and map information of traffic participants, and proposes a hierarchical encoding scheme to improve local and global interactions in the observation scene. EEGT[2] further proposed an edge-enhanced graph transformer to handle diverse interactions in traffic scenes. However, these methods focus on modeling and representing interactions in the observation scene, while ignoring the interaction modeling of future movements.

[0006] In addition, some research works implicitly model future interactions through attention mechanisms between agents. For example, SceneTransformer[3] makes joint predictions in the decoder through alternating attention layers across time and agents. MTR++[4] further considers the relative spatial relationship of each pair of agent intent queries in the attention layer to make scenario-appropriate predictions for multiple agents. Another approach, TrafficSim[5], uses latent variable models to implicitly represent the uncertainty and interactions in future scenarios. In addition, some research works use marginal predictions to generate joint predictions. For example, THOMAS[6] uses an attention mechanism to recombine the candidate marginal predictions of each agent to obtain a consistent joint prediction. Similarly, JFP[7] constructs an interaction graph of agents based on the proximity of the marginal candidate predictions of each pair of agents and uses a message passing mechanism to approximate the joint probability. FFINet[8] further proposes a feedback module to handle the impact of future interactions on the initial prediction.

[0007] On the other hand, some methods use conditional models for joint prediction. For example, M2I[9] first identifies the influencers and reactants between two agents, and then predicts the future motion of the reactant based on the marginal prediction of the influencer. However, it only considers the interaction between the two agents. FJMP

[10] further utilizes directed acyclic graphs to model the interactions between any number of agents and uses graph neural networks for marginal and conditional reasoning. However, these models are still limited to using static graphs to handle future interactions, which will continue to evolve as the agents move forward, especially in tasks with long prediction horizons. Existing technologies only consider the interactions between traffic participants in future scenarios, while ignoring the dynamic changing characteristics of these interactions. At the same time, existing joint prediction decoding schemes ignore the use of predicted future information of multi-agent joint motion to improve future scenario modeling, which can easily lead to error accumulation and affect the accuracy and consistency of long-term joint predictions. Summary of the Invention

[0008] The purpose of the present invention is to overcome the problem of poor accuracy and consistency of long-term joint prediction caused by using static images and to provide a multi-agent joint motion prediction method based on dynamic images and progressive multi-scale decoding.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding, the method comprising the following steps:

[0011] Set up n agents corresponding to n surrounding traffic participants in the same traffic scene, obtain the historical state of the agent at the time step, input the historical state into the scene encoder, and output the agent features. The scene encoder consists of an agent encoder, a road graph convolutional network, and a fusion graph convolutional network;

[0012] The agent features are input into the decoder, which includes a timing module, a dynamic graph progressive construction module, a heterogeneous graph convolution module, and a prediction tap. The prediction tap includes a coarse prediction tap and a joint prediction tap. The decoder outputs the joint future state of multiple agents. Each joint future state is called a mode. The traffic scene is changed and the above steps are repeated to obtain the joint future state of the agents in K different scenes, that is, K modes. The loss function is calculated based on the joint future state to train the scene encoder and decoder to obtain a motion prediction model. The motion prediction model obtains the actual agent features and outputs the agent joint motion prediction result.

[0013] The specific steps of inputting the agent features into the decoder and outputting the joint future state of multiple agents are as follows:

[0014] The agent features are input into the time series module of the decoder and converted into a two-dimensional tensor. The two-dimensional tensor consists of latent features. Position embedding is applied to the latent features. The two-dimensional tensor after position embedding is subjected to the cross-time attention mechanism to obtain the T of n agents. f The attention features of time steps, where T f Indicates the prediction time range. The dynamic graph progressive construction module uses dynamic graphs to model and represent future scenarios. Every τ time step, a time snapshot of the dynamic graph is used to represent the traffic scenario within τ time steps. For the entire prediction range T f , the dynamic graph consists of P time snapshots, and the nodes in the graph are divided into agent nodes and lane segment nodes, where T f =τ*P, and for p=1,2,…,P, perform the following steps:

[0015] The attention features of n agents from the τ*(p-1)+1th to the τ*pth time step are input into the dynamic graph progressive construction module, which constructs a snapshot based on the attention features and the complete joint state of the agents at the τ*(p-1)th time step in and ε p Represents snapshots The vertex set and edge set of the snapshot, the attention feature is used as the feature of the agent node of the snapshot, and the heterogeneous graph convolution module is based on the snapshot. The first agent node updates the feature z i，p, where i = 1, 2, ..., n, i represents the serial number of the agent, and the rough prediction tap is based on the agent node update feature z i，p Predict the positions of n agents at the final moment τ*p from the τ*(p-1)+1th to the τ*pth time step, obtain the rough future state of the agent at the τ*pth moment, update the snapshot based on the rough future state of the agent at the τ*pth moment, and obtain the updated snapshot Heterogeneous graph convolution module based on updated snapshot Generate the second agent node to update the feature z′ i，p , the joint prediction tap updates the feature z′ according to the second agent node i，p , output the complete joint state of the agent from the τ*(p-1)+1th to the τ*pth time step

[0016] Furthermore, the agent features are:

[0017] h i，k =h i +m k , i=1,2,...,n

[0018] Among them, h i,k represents the agent characteristics of the i-th agent under the k-th modality, h i represents the agent characteristics of the i-th agent, m k represents the learnable contextual embedding of the k-th modality, where each modality corresponds to a scene.

[0019] Furthermore, all states of the agent include the agent's position, speed, heading angle, shape, and agent type, and all states include a joint predicted state, a historical state, and a rough future state.

[0020] Furthermore, the first agent node updates the feature z i，p for:

[0021]

[0022] in, For snapshots Any node in exist There are two kinds of neighborhoods, namely agent neighborhoods and lane neighborhood Snapshot The characteristics x of the agent node in ·,p , snapshot The agent node coordinates c in ·，p ,m(·) is the snapshot The feature x corresponding to the adjacent node pair i and ji，p and x j，p The message calculated from the node coordinates, i, j and l represent different agent nodes, f a and f l is the function approximated by the multilayer perceptron, and g represents the self-transformation achieved by the multilayer perceptron.

[0023] Furthermore, the second agent node updates the feature based on the updated snapshot The node features and node coordinates are calculated in the same way as the first agent node update features.

[0024] Further, build snapshot The specific steps are:

[0025] Based on the complete joint state, the connection relationship between the nodes of the snapshot is constructed. The nodes of the snapshot are divided into agent nodes and lane segment nodes. Then, a fully connected graph is constructed between the agent nodes and the agent nodes. The connection between lane segment nodes is established based on the road topology. The distance between each agent node and its surrounding lane segment nodes is calculated based on the complete joint state. When the distance is less than the hyperparameter ε, an edge is constructed between the corresponding agent node and the lane segment node.

[0026] Furthermore, the loss function is:

[0027]

[0028] in, represents the joint prediction loss, represents the loss of coarse-grained prediction, represents the auxiliary loss corresponding to the marginal prediction, and λ1 and λ2 are hyperparameters used to balance these three objectives.

[0029] Furthermore, the joint prediction loss is:

[0030]

[0031] in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the joint predicted state in the k*th modality, T f represents the forecast time range, Represents the set of all agent nodes, l represents smooth Norm loss, y i，t and denote the true state of the i-th agent at time t and the joint predicted state of agent i in the k*-th modality, respectively.

[0032] Furthermore, the auxiliary loss corresponding to the marginal prediction is:

[0033]

[0034] in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the marginal predicted state in the k*th mode, y i，t and The table represents the true state of the i-th agent at time t and the marginal predicted state of agent i in the k*-th mode, where the marginal predicted state is the output of the marginal prediction tap after the latent feature is input into the marginal prediction tap.

[0035] Furthermore, the loss of coarse-grained prediction is:

[0036]

[0037] in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the rough future state in the k*th mode, T f represents the forecast time range, Represents the set of all agent nodes, l represents smooth Norm loss, y i，t and The table represents the true state of the i-th agent at time t and the rough future state of agent i in the k*-th mode, which is the output of the rough prediction tap after the latent feature input rough prediction tap, and P is the number of static snapshots in the dynamic graph.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention further considers the evolution of future interactions by modeling future scenarios using dynamic graphs rather than static graphs. By continuously capturing and utilizing the surrounding environment and interactions, the present invention uses a progressive decoding strategy. This strategy involves first making a rough prediction of the agent node at the current time step to obtain a rough future state. Based on this rough state, the snapshot is updated, i.e., the traffic scene is updated. The joint future state is then predicted. By gradually unfolding the future scenario, the dynamic spatiotemporal information of future motion is effectively processed. Through multi-scale predictions from coarse to fine granularity, the predicted results are used to update the scenario modeling representation, rather than relying solely on the representation of the observed scenario. This reduces the uncertainty of multiple interactions in the future scenario based on existing predictions, thereby better improving the accuracy of the prediction and the consistency of the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] This paper proposes a novel progressive multi-scale decoding strategy, called ProgD, and a multi-agent joint motion prediction method using ProgD, enabled by dynamic heterogeneous graph-based scene modeling. Specifically, considering the inherent uncertainty of future interactions, a dynamic heterogeneous graph-based progressive scene modeling method is designed to explicitly and comprehensively capture the evolving interactions in future scenarios. As this dynamic heterogeneous graph unfolds, a spatiotemporal decomposition architecture is designed to handle spatiotemporal dependencies in future scenarios and gradually eliminate uncertainty in the future motion of multiple traffic participants. Furthermore, a multi-scale decoding process is incorporated to improve future scene modeling and consistent prediction of future participant motion. The proposed ProgD achieves state-of-the-art performance on both the INTERACTION Multi-Agent Prediction Benchmark and the Argoverse 2 Multi-World Prediction Benchmark. This paper proposes an innovative progressive modeling strategy that leverages a dynamic heterogeneous graph to explicitly encode the diverse interactions between agents and the road network in future scenarios, taking into account the dynamic evolution of these interactions over time. At the same time, the present invention proposes a progressive multi-scale decoding strategy. By unfolding the dynamic graph, ProgD gradually eliminates the inherent uncertainty in the future interactive motion of multiple intelligent agents, and combines the multi-scale decoding process to alternately improve the complex interaction modeling in future scenarios to improve the accuracy and consistency of the prediction.

[0043] The goal of this invention is to jointly predict the n traffic participants in a scene in the next T seconds based on the historical status of the traffic participants in the past T0 seconds and the road network information. f For each scene, K joint predictions are made for the key traffic participants in the scene, and each joint prediction is called a modality. To achieve this goal, an encoder-decoder framework is adopted. The encoder encodes the historical information of traffic participants and the road network, while the decoder predicts K joint movements of multiple key traffic participants in the scene in the future, such as Figure 1 shown.

[0044] 1. Scene coding:

[0045] The input information of each scene includes the historical states of all agents in the scene and the road network. The historical state s of agent i at time step t is i，t This includes its position, speed, heading angle, shape, agent type, etc. The road network is represented by polylines, which are composed of sampled lane center points. To facilitate information dissemination between multiple agents, a scene-centered coordinate system is used. All scene elements are represented in the global coordinate system.

[0046] The scene encoder consists of an agent encoder, a road graph convolutional network, and a fusion graph convolutional network. The agent encoder encodes the historical motion features of each agent. The road graph convolutional network generates features for each lane segment. Based on the above agent features and lane segment features, the fusion graph convolutional network processes agent-agent and agent-lane segment interactions in the observed scene using graph convolution. The final output is the representation of each agent [h1, h2, ..., h n ] as the agent feature.

[0047] 2. Joint Motion Prediction Based on Progressive Multi-Scale Decoding

[0048] 2.1 Progressive Modeling of Future Scenarios

[0049] As the agent moves forward, its interactions with surrounding vehicles, pedestrians, and roads undergo dynamic changes. A discrete-time dynamic heterogeneous graph is used to structurally represent the evolving and diverse interactions in the scene. Nodes correspond to different scene elements, such as agents and lane segments, while different types of edges represent various interactions, including interactions between agents and between agents and lanes. The attribute information of nodes and the connections between nodes continue to change as the dynamic scene unfolds. However, the future state of multiple agents is inherently unknown and uncertain. To address this challenge, a step-by-step scenario modeling approach is adopted, where the dynamic graph structure at the next time step is constructed based on the joint future state of multiple agents decoded previously.

[0050] Specifically, the future scenario is represented as a discrete-time dynamic graph It consists of a set of discrete snapshots Composition, of which and ε p Represents snapshots The vertex and edge sets of . Sequentially model the future scene information of each τ time step, with a prediction range of T f =τ*P. In particular, the scene interaction within time step t∈[τ*(p-1)+1,τ*p] is represented by the graph The topological structure of is encoded, p = 1, 2, ..., P. Two types of nodes are considered, namely agent nodes and lane segment nodes. The interactions between agents and lane segments are modeled by different types of edges. The next initial snapshot This is constructed based on each agent's predicted state and lane characteristics at time t = τ*(p-1). Specifically, a fully connected graph is constructed between agents in each scenario. Lane segment nodes are connected based on the road topology. Furthermore, the distance between each agent and its surrounding lane segments is considered to construct dynamic lane-agent edges. An edge is constructed between corresponding node pairs if and only if this distance is less than the hyperparameter ε.

[0051] As described above, the next dynamic snapshot of the dynamic graph is constructed based on the prediction of the previous time period, which in turn provides the basis for the prediction of the subsequent time period. This progressive scenario modeling approach can capture the dynamic evolution of the future scenario.

[0052] 2.2 Multi-scale Decoding

[0053] To address potential error accumulation in the progressive prediction framework, a multi-scale decoding scheme is introduced. This scheme performs motion prediction by alternating between updating the dynamic image snapshot and refining the prediction state. Specifically, the prediction process consists of three steps: coarse-grained prediction based on dynamic image information, updating the dynamic image snapshot based on the coarse-grained prediction, and joint prediction.

[0054] Rough prediction: based on the constructed dynamic graph snapshot The features of the agent are used to jointly predict the coarse-grained prediction of the agent in the next τ time steps, that is, only the final position of multiple agents in the time period t∈[τ*(p-1)+1,τ*p] is predicted.

[0055] Snapshot update: Update the graph snapshot based on the agent’s future state at time t = τ * p (instead of the state at time t = τ * (p-1) used in snapshot construction) The properties and connections of the agent nodes in

[0056] Joint Forecasting: Leveraging Updated Snapshots The joint prediction tap finally outputs the entire joint state of the agent in the time period t∈[τ*(p-1)+1,τ*p] Where i = 1, 2, ..., n. Repeat this process until t = T f .

[0057] 2.3 Network Architecture

[0058] like Figure 1 As shown in Figure 2, the decoder consists of a temporal module, a dynamic graph progressive construction and update module, two heterogeneous graph convolution modules for processing the two-stage dynamic graph, and several taps responsible for decoding the joint future state of the interacting agents in the scene. The following describes the main components in detail.

[0059] Specifically, a spatiotemporal decomposition strategy is adopted to process the spatiotemporal information in dynamic graphs. Based on the agent features extracted from the encoder, the temporal features corresponding to the future motion of the agent are first decoded. In order to promote smooth and coherent motion trajectories, a cross-temporal attention mechanism is adopted to improve the temporal consistency between the features of each agent within the prediction range. In addition, the designed heterogeneous graph convolution module is used to process the spatial interactions of each future time step. With the help of message passing in heterogeneous graph convolution, features from different nodes (corresponding to agents and lanes) are continuously aggregated and transformed to obtain spatially consistent future features of multiple agents. Finally, based on these agent features that are consistent in both space and time, the prediction tap outputs the joint future state of multiple agents.

[0060] Temporal module. In order to improve the temporal consistency of future movements, a cross-temporal attention mechanism is used to process the agent features h extracted from the encoder. i , where i = 1, 2, ..., n. Specifically, in order to facilitate the subsequent modules to process the future interactions between the agent and the surrounding road network, a set of latent variables are introduced to represent the characteristics of the agent at different future time steps. Specifically, each agent feature It is converted into a two-dimensional tensor through a one-dimensional convolution layer. This corresponds to agent i in the future T f In addition, the cross-temporal attention mechanism is used to process their respective latent features x i，t , where t=1,2,…,T f , to capture the time dependency of each agent’s future motion. i，t A position embedding is applied to provide the position information of time step t. The position embedding can be a learnable parameter or a predefined function such as a combination of sine and cosine functions.

[0061] x i,t =x i，t +PE(x i，t )

[0062] x i =Attention(x i ).

[0063] Heterogeneous graph convolution operator. The designed graph convolution operator is based on a message passing scheme to aggregate information from agent nodes and lane nodes along different types of edges. Specifically, given a dynamic graph snapshot For any agent node First, in its two neighborhoods (agent neighborhood and lane neighborhood ) propagates the feature x based on the node pair ·，p and node coordinates c ·，p The message m(·) is calculated and aggregated using max pooling. Then, the aggregated features are respectively passed through the function f approximated by the multi-layer perceptron (MLP) a (·) and f l (·) is transformed and compared with the self-transformation feature g(x i，p ) are added to update the node where g(·) represents the self-transformation achieved by a multi-layer perceptron (MLP).

[0064]

[0065] Similarly, the graph convolution operator in the second graph convolution module is updated with the dynamic graph snapshot As input, it generates agent node features z′ i，p .

[0066] Prediction taps. All prediction taps are designed as multi-layer perceptrons. Specifically, the coarse-grained prediction taps are based on the agent feature z output by the first heterogeneous graph convolution module. i，p , i=1,2,…,n, predict the position of multiple agents at the final moment in the next τ time steps (t∈[τ*(p-1)+1,τ*p]). The joint prediction tap is based on the snapshot updated by the second heterogeneous graph convolution module Agent feature z′ in i，p , and output the complete joint state of the agent at t∈[τ*(p-1)+1,τ*p] In addition, to facilitate the time series module to effectively capture the temporal information of the agent's future motion, a marginal prediction tap is introduced. It predicts the marginal distribution of each agent's future state based on the agent's latent features.

[0067] Multimodal prediction. Finally, given the uncertainty of agent intentions and movements, K predictions are made for the joint movement of multiple agents (each joint prediction is called a modality). In order to encourage the model to make consistent predictions within each modality, a learnable context embedding m is introduced. k, where k = 1, 2, ..., K, represents the scene potential features of each modality. In each modality, the agent features obtained from the encoder are combined with the corresponding scene embedding. This approach allows the model to utilize the individual features of each agent while taking advantage of the overall context information provided by the scene embedding. Therefore, the predictions of all agents in the same modality undergo a bias process. Mathematically speaking, the agent features h in the encoder are i becomes

[0068] h i，k =h i +m k , i=1,2,...,n

[0069] The subscript k indicates that the variables belong to different modes.

[0070] 2.4 Training

[0071] The training goal of the model is to minimize the joint prediction of the future state of the agent and the corresponding true value y i,t Smoothing between Norm loss. This loss is calculated based on the best mode among k=1,2,…,K predictions in terms of joint endpoint offset error metric. Mathematically,

[0072]

[0073] in, and Represents time T f The true position of the agent node at time t and the predicted position in the k*th mode. Similarly, the loss of coarse-grained prediction is calculated and recorded as

[0074] In addition, the auxiliary loss corresponding to the marginal prediction is calculated in a similar way, except that each agent uses its own best marginal prediction.

[0075] The encoder-decoder network optimizes the weighted sum of these losses via gradient descent, where λ1 and λ2 are hyperparameters used to balance these three objectives.

[0076]

[0077] During the training process of the present invention, the steps for generating the joint state are as follows:

[0078] The attention features of the n agents from the 1st to the τth time step are input into the dynamic graph progressive construction module, which constructs a snapshot based on the predicted state of the agent at the 0th time step (the predicted state of the agent at the 0th time step is the initialized known quantity, which can be the state of the agent at the last moment of the historical data) and the lane features Attention features are used as snapshot agent node features;

[0079] Heterogeneous graph convolution module based on snapshot The first agent node updates the feature based on the feature calculation, and the rough prediction tap predicts the next τ time steps, that is, the position of the n agents at the final moment from the 1st to the τth time step, based on the updated feature of the agent node. This also obtains the rough future state of the agent at time t = τ. Based on the rough future state at time t = τ, the snapshot is updated to obtain the updated snapshot The heterogeneous graph convolution module generates the updated features of the second agent node based on the updated snapshot, and the joint prediction tap outputs the joint state of the agents from the 1st to the τth time step according to the updated features of the second agent node;

[0080] The attention features of n agents from the τ+1th to the 2τth time step are input into the dynamic graph progressive construction module, which constructs a snapshot based on the joint state of the τth time step and the lane features. Heterogeneous graph convolution module based on snapshot The first agent node update feature is calculated based on the feature of the agent node, and the coarse prediction tap predicts the rough future state (which includes position information) of the n agents at the final moment in the next τ time steps, that is, the τ+1th to 2τth time steps, based on the agent node update feature. The snapshot is updated based on the rough future state of the agent at time t=2τ, and the updated snapshot is obtained. The heterogeneous graph convolution module generates the second agent node update feature based on the updated snapshot. The joint prediction tap outputs the joint state of the agent from the τ+1th to the 2τth time steps according to the second agent node update feature; and so on until the time step T is reached. f .

[0081] The beneficial effects of the present invention are:

[0082] (1) The proposed method further considers the evolution of future interactions by adopting dynamic graphs rather than static graphs to model future scenes. By continuously capturing and utilizing the surrounding environment and interactions, ProgD gradually unfolds the future scene using a progressive decoding strategy, effectively processing the dynamic spatiotemporal information of future motion.

[0083] (2) Through multi-scale prediction from coarse-grained to fine-grained, ProgD updates the dynamic graph of the optimized model based on the coarse-grained prediction results to improve the accuracy of modeling future scenarios, thereby improving the consistency and accuracy of multi-agent motion prediction.

[0084] Numerical experiments have verified that the proposed framework significantly improves the accuracy and consistency of prediction results compared to existing joint motion prediction methods. On the INTERACTION Multi-Agent joint motion prediction benchmark dataset, the proposed method achieves excellent performance in terms of metrics such as minimum joint average displacement error (minJADE), minimum joint final displacement error (minJFDE), ego collision rate (egoCR), and consistent minimum joint miss rate (consistent minimum joint miss rate (consist-minJMR)). Furthermore, on the Argoverse2 Multi-World prediction benchmark dataset, the proposed method also performs well in terms of metrics such as minimum joint average displacement error (minJADE), minimum joint final displacement error (minJFDE), and actor miss rate (actorMR).

[0085] Figure 1 The encoder encodes the observed trajectories of traffic participants (agents) and the road network in the scene and outputs the features h1, h2, ..., h n The decoder consists of an inter-temporal module, two graph convolutional modules for processing two-stage dynamic graphs, and several taps responsible for decoding the marginal and joint future states of traffic participants. The inter-temporal module is designed to handle the temporal dependencies in the future motion of each agent, while the two graph convolutional modules process the dynamic graph that structuredly represents the dynamic interactions in the future motion of multiple agents.

[0086] The taps of the present invention can adopt a common prediction network.

[0087] References

[0088] [1]Zhou,Z.,Ye,L.,Wang,J.,Wu,K.,Lu,K.,2022.HiVT:Hierarchical vectortransformer for multi-agent motion prediction,in:Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,New Orleans,LA,USA.pp.8813–8823.

[0089] [2]Zhang,T.,Fu,M.,Yang,Y.,Song,W.,Liu,T.,2024.Edge-enriched graphtransformer for multi-agent trajectory prediction with relative positionalsemantics.IEEE Transactions on Instrumentation and Measurement 73.

[0090] [3]Ngiam,J.,Vasudevan,V.,Caine,B.,Zhang,Z.,Chiang,H.T.L.,Ling,J.,Roelofs,R.,Bewley,A.,Liu,C.,Venugopal,A.,et al.,2022.Scene trans-former:Aunified architecture for predicting future trajectories of multiple agents,in:The 10th International Conference on Learning Representations.

[0091] [4]Shi,S.,Jiang,L.,Dai,D.,Schiele,B.,2024.MTR++:Multi-agent motionprediction with symmetric scene modeling and guided intention querying.IEEETransactions on Pattern Analysis and Machine Intelligence 46,3955–3971.

[0092] [5]Suo,S.,Regalado,S.,Casas,S.,Urtasun,R.,2021.TrafficSim:Learning tosimulate realistic multi-agent behaviors,in:Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,virtual.pp.10400–10409.

[0093] [6]Gilles,T.,Sabatini,S.,Tsishkou,D.,Stanciulescu,B.,Moutarde,F.,2022b.THOMAS:Trajectory heatmap output with learned multi-agent sampling,in:The 10th International Conference on Learning Representations,Virtual Event.

[0094] [7]Luo,W.,Park,C.,Cornman,A.,Sapp,B.,Anguelov,D.,2022.JFP:Jointfuture prediction with interactive multi-agent modeling for autonomousdriving,in:Conference on Robot Learning,Auckland,New Zealand.pp.1457–1467.

[0095] [8]Kang,M.,Wang,S.,Zhou,S.,Ye,K.,Jiang,J.,Zheng,N.,2024.FFINet:Futurefeedback interaction network for motion forecasting.IEEE Transactions onIntelligent Transportation Systems.

[0096] [9]Sun, Q., Huang,

[0097]

[10] Rowe, L., Ethier, M., Dykhne, EH, Czarnecki, K., 2023. FJMP: Factorizedjoint multi-agent motion prediction over learned directed acyclic interaction graphs, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.pp.13745–13755.

[0098] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding, characterized by: The method comprises the following steps: Set up n agents corresponding to n surrounding traffic participants in the same traffic scene, obtain the historical state of the agent at the time step, input the historical state into the scene encoder, and output the agent features. The scene encoder consists of an agent encoder, a road graph convolutional network, and a fusion graph convolutional network; The agent features are input into the decoder, which includes a timing module, a dynamic graph progressive construction module, a heterogeneous graph convolution module, and a prediction tap. The prediction tap includes a coarse prediction tap and a joint prediction tap. The decoder outputs the joint future state of multiple agents. Each joint future state is called a mode. The traffic scene is changed and the above steps are repeated to obtain the joint future state of the agents in K different scenes, that is, K modes. The loss function is calculated based on the joint future state to train the scene encoder and decoder to obtain a motion prediction model. The motion prediction model obtains the actual agent features and outputs the agent joint motion prediction result. The specific steps of inputting the agent features into the decoder and outputting the joint future state of multiple agents are as follows: The agent features are input into the time series module of the decoder and converted into a two-dimensional tensor. The two-dimensional tensor consists of latent features. Position embedding is applied to the latent features. The two-dimensional tensor after position embedding is subjected to the cross-time attention mechanism to obtain the T of n agents. f The attention features of time steps, where T f Indicates the prediction time range. The dynamic graph progressive construction module uses dynamic graphs to model and represent future scenarios. Every τ time step, a time snapshot of the dynamic graph is used to represent the traffic scenario within τ time steps. For the entire prediction range T f , the dynamic graph consists of P time snapshots, and the nodes in the graph are divided into agent nodes and lane segment nodes, where T f =τ*P, and for p=1,2,…,P, perform the following steps: The attention features of n agents from the τ*(p-1)+1th to the τ*pth time step are input into the dynamic graph progressive construction module, which constructs a snapshot based on the attention features and the complete joint state of the agents at the τt(p-1)th time step in and ε p Represents snapshots The vertex set and edge set of the snapshot, the attention feature is used as the feature of the agent node of the snapshot, and the heterogeneous graph convolution module is based on the snapshot. The first agent node updates the feature of the feature calculation Where i = 1, 2, ..., n, i represents the serial number of the agent, and the rough prediction tap is based on the updated features of the agent node. Predict the positions of n agents at the final moment τ*p from the τ*(p-1)+1th to the τ*pth time step, obtain the rough future state of the agent at the τ*pth moment, update the snapshot based on the rough future state of the agent at the τ*pth moment, and obtain the updated snapshot Heterogeneous graph convolution module based on updated snapshot Generate the second agent node update feature The joint prediction tap updates the features based on the second agent node Output the complete joint state of the agent from the τ*(p-1)+1th to the τ*pth time step i=1,2,…,n。 2. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 1, characterized in that: The agent features are: h i，k =h i +m k ,i=1,2,...,n Among them, h i,k represents the agent characteristics of the i-th agent under the k-th modality, h i represents the agent characteristics of the i-th agent, m k represents the learnable contextual embedding of the k-th modality, where each modality corresponds to a scene.

3. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 2, characterized in that: All states of the agent include the agent's position, speed, heading angle, shape and agent type, and all states include the joint predicted state, historical state and rough future state.

4. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 1, characterized in that: The first agent node updates the feature z i，p for: in, For snapshots Any node in exist There are two kinds of neighborhoods, namely agent neighborhoods and lane neighborhood Snapshot The features x.,p,snapshot of the agent node in The coordinates of the agent node c., p, m(·) in the snapshot are The feature x corresponding to the adjacent node pair i and j i，p and x j，p The message calculated from the node coordinates, i, j and l represent different agent nodes, f a and f l is the function approximated by the multilayer perceptron, and g represents the self-transformation achieved by the multilayer perceptron.

5. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 4, characterized in that: The second agent node updates features based on the updated snapshot The node features and node coordinates are calculated in the same way as the first agent node update features.

6. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 1, characterized in that: Build Snapshot The specific steps are: Based on the complete joint state, the connection relationship between the nodes of the snapshot is constructed. The nodes of the snapshot are divided into agent nodes and lane segment nodes. Then, a fully connected graph is constructed between the agent nodes and the agent nodes. The connection between lane segment nodes is established based on the road topology. The distance between each agent node and its surrounding lane segment nodes is calculated based on the complete joint state. When the distance is less than the hyperparameter ε, an edge is constructed between the corresponding agent node and the lane segment node.

7. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 1, characterized in that: The loss function is: in, represents the joint prediction loss, represents the loss of coarse-grained prediction, represents the auxiliary loss corresponding to the marginal prediction, and λ1 and λ2 are hyperparameters used to balance these three objectives.

8. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 7, characterized in that: The joint prediction loss is: in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the joint predicted state in the k*th modality, T f represents the forecast time range, Represents the set of all agent nodes, l represents smooth Norm loss, y i，t and denote the true state of the i-th agent at time t and the joint predicted state of agent i in the k*-th modality, respectively.

9. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 8, characterized in that: The auxiliary loss corresponding to the marginal prediction is: in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the marginal predicted state in the k*th mode, y i，t and The table represents the true state of the i-th agent at time t and the marginal predicted state of agent i in the k*-th mode, where the marginal predicted state is the output of the marginal prediction tap after the latent feature is input into the marginal prediction tap.

10. The multi-agent joint motion prediction method based on dynamic graph and progressive multi-scale decoding according to claim 9, characterized in that: The loss of coarse-grained prediction is: in, and Represents the time step T f The actual position of the node corresponding to the agent and the position in the rough future state in the k*th mode, T f represents the forecast time range, Represents the set of all agent nodes, l represents smooth Norm loss, y i，t and The table represents the true state of the i-th agent at time t and the rough future state of agent i in the k*-th mode, which is the output of the rough prediction tap after the latent feature input rough prediction tap, and P is the number of static snapshots in the dynamic graph.

Images