An agent trajectory prediction method and system

Abstract

The application discloses an intelligent body trajectory prediction method and system, and belongs to the technical field of intelligent body trajectory prediction. The method comprises the following steps: acquiring a current scene video, and constructing a dynamic space-time graph of intelligent bodies in the current scene; identifying the dynamic space-time graph to obtain trajectory distribution information of the intelligent bodies; identifying the trajectory distribution information of the intelligent bodies through a terminal driving module to obtain a future trajectory terminal point predicted by the intelligent bodies; inputting the trajectory distribution information and the predicted future trajectory terminal point into a forward decoder after connection to obtain a forward hidden state; inputting the trajectory distribution information into a backward decoder to obtain a backward hidden state; connecting the forward hidden state and the backward hidden state to predict a future trajectory of the intelligent bodies and obtain a trajectory prediction result. The accuracy of intelligent body trajectory prediction is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-target trajectory prediction technology in complex traffic scenarios, and in particular to an intelligent agent trajectory prediction method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As society moves towards greater intelligence and automation, the perception, understanding, and prediction of the movement trajectories of intelligent agents in a given scenario becomes increasingly important. Specifically, predicting the future position of dynamic intelligent agents and planning accordingly is a key task in fields such as autonomous vehicles, intelligent urban monitoring systems, and mobile robot navigation. In autonomous driving, accurate prediction of agent trajectories enables controllers to plan vehicle movement in advance, effectively avoiding collisions and preventing traffic accidents. Intelligent urban monitoring systems with real-time decision-making capabilities must understand the movement routes of intelligent agents in a scene to help identify suspicious activities and issue early warnings. Socially aware mobile robots need to accurately predict the intentions of surrounding intelligent agents during navigation to plan reasonable paths.

[0004] However, accurately predicting the movement of intelligent agents is highly challenging due to the complexity of their behavior and the diversity of internal and external factors. First, the trajectory of a single agent depends on other agents in the surrounding environment, making it difficult to model the interactions between agents. Second, ensuring that the predicted future trajectory conforms to both physical constraints and social norms is another challenge. An agent's behavior depends not only on pedestrians and vehicles but also heavily on the surrounding physical environment; for example, fixed obstacles on roads, sidewalks, and grass can influence decision-making. Furthermore, agent behavior is constrained by social rules; some trajectories may be physically feasible but socially unacceptable. Finally, the movement of intelligent agents is multimodal. In real-world scenarios, generating a deterministic trajectory is unrealistic; multiple reasonable and feasible trajectories often exist.

[0005] Traditional trajectory prediction models, such as Kalman filtering, social force models, continuum dynamics models, and Gaussian process dynamics models, are based on handcrafted functions and rules, which limit their ability to learn knowledge outside the domain. They also often fail to simulate the movement of intelligent agents well in complex and crowded environments, and accumulate errors as the prediction time increases. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes an intelligent agent trajectory prediction method and system, which solves the issues of insufficient modeling of intelligent agent interaction information and poor long-term prediction performance in complex traffic scenarios in existing technologies, thereby improving the accuracy of trajectory prediction for intelligent agents such as pedestrians and vehicles.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Firstly, a method for predicting the trajectory of an intelligent agent is proposed, including:

[0009] Acquire video of the current scene and construct a dynamic spatiotemporal graph of the agents in the current scene;

[0010] By identifying dynamic spatiotemporal graphs, the trajectory distribution information of intelligent agents can be obtained;

[0011] The endpoint-driven module identifies the trajectory distribution information of the agent and obtains the predicted future trajectory endpoint of the agent.

[0012] Based on trajectory distribution information, predicted future trajectory endpoints, and a decoder, the agent's future trajectory is predicted to obtain a trajectory prediction result. The decoder includes a forward decoder and a backward decoder. The trajectory distribution information and the predicted future trajectory endpoints are concatenated and input into the forward decoder to obtain a forward hidden state. The trajectory distribution information is input into the backward decoder to obtain a backward hidden state. The forward hidden state and the backward hidden state are concatenated to predict the agent's future trajectory to obtain a trajectory prediction result.

[0013] Secondly, an intelligent agent trajectory prediction system is proposed, including:

[0014] The dynamic spatiotemporal graph acquisition module is used to acquire the current scene video and construct the dynamic spatiotemporal graph of the agent in the current scene;

[0015] The trajectory distribution information acquisition module is used to identify dynamic spatiotemporal graphs and obtain the trajectory distribution information of the intelligent agent;

[0016] The future trajectory endpoint prediction module is used to identify the trajectory distribution information of the agent through the endpoint driving module and obtain the predicted future trajectory endpoint of the agent.

[0017] The future trajectory prediction module is used to predict the future trajectory of the agent based on trajectory distribution information, the predicted future trajectory endpoint, and the decoder, and obtain the trajectory prediction result. The decoder includes a forward decoder and a backward decoder. The trajectory distribution information and the predicted future trajectory endpoint are connected and then input into the forward decoder to obtain the forward hidden state. The trajectory distribution information is input into the backward decoder to obtain the backward hidden state. The forward hidden state and the backward hidden state are connected to predict the future trajectory of the agent and obtain the trajectory prediction result.

[0018] Thirdly, an electronic device is proposed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps described in an intelligent agent trajectory prediction method.

[0019] Fourthly, a computer-readable storage medium is proposed for storing computer instructions, which, when executed by a processor, complete the steps described in the intelligent agent trajectory prediction method.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention considers the randomness of agent movement and the dynamic nature of the number of agents in a scene, and applies dynamic spatiotemporal graphs to the trajectory prediction problem, effectively modeling the social interaction relationships between agents; it introduces an endpoint-driven module, which first predicts the endpoint of the agent's movement, and then predicts intermediate trajectory points based on the endpoint, providing a good guide for trajectory prediction; it designs a decoder based on reverse enhancement decoding, which improves long-term prediction performance by fusing reverse and forward hidden states; and it uses various types of loss functions to optimize model parameters, enabling more accurate learning of the agent's movement trends and improving the accuracy of trajectory prediction.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0024] Figure 1 This is a flowchart of the method disclosed in Example 1;

[0025] Figure 2 This is a diagram showing the overall structure of the intelligent agent trajectory prediction network disclosed in Example 1;

[0026] Figure 3A visualization of the multimodal trajectories predicted in Example 1;

[0027] Figure 4 This is a visualization of the trajectory distribution predicted in Example 1. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Example 1

[0032] To address the shortcomings of existing technologies in modeling intelligent agent interaction information in complex traffic scenarios and their poor long-term prediction performance, and to improve the accuracy of trajectory prediction for intelligent agents such as pedestrians and vehicles, this embodiment discloses an intelligent agent trajectory prediction method, such as... Figure 1 As shown, it includes:

[0033] Acquire video of the current scene and construct a dynamic spatiotemporal graph of the agents in the current scene;

[0034] By identifying dynamic spatiotemporal graphs, the trajectory distribution information of intelligent agents can be obtained;

[0035] The endpoint-driven module identifies the trajectory distribution information of the agent and obtains the predicted future trajectory endpoint of the agent.

[0036] Based on trajectory distribution information, predicted future trajectory endpoints, and a decoder, the agent's future trajectory is predicted to obtain a trajectory prediction result. The decoder includes a forward decoder and a backward decoder. The trajectory distribution information and the predicted future trajectory endpoints are concatenated and input into the forward decoder to obtain a forward hidden state. The trajectory distribution information is input into the backward decoder to obtain a backward hidden state. The forward hidden state and the backward hidden state are concatenated to predict the agent's future trajectory to obtain a trajectory prediction result.

[0037] This embodiment provides a detailed description of an intelligent agent trajectory prediction method.

[0038] In order to achieve accurate prediction of the agent's trajectory, this embodiment constructs, as follows: Figure 2 The agent trajectory prediction network model shown includes an agent motion trajectory extraction module, an encoder module, an endpoint driving module, and a decoder module.

[0039] By acquiring the motion trajectories of each agent in existing scene videos and constructing a dataset, the agent trajectory prediction network model is trained and tested using the dataset, thereby obtaining a trained agent trajectory prediction network model, which is used to identify the current scene video and predict the future trajectory of the agent in the current scene video.

[0040] The process of obtaining the motion trajectories of each agent in an existing scene video and constructing a dataset through the agent motion trajectory extraction module is as follows:

[0041] The DJI Matrice 200 quadcopter drone was selected as the data acquisition device. This model integrates a Zenmuse X5S gimbal camera, capable of shooting high-definition video at 3840x2160 resolution at 30 frames per second. Data acquisition locations were selected within the scenarios requiring prediction. Taking Jinan, Shandong Province as an example, footage was taken above several intersections of different shapes and traffic densities. The video length for each intersection was approximately 8-15 minutes. The diverse scenarios captured by the drone included urban traffic scenes such as large square intersections, small square intersections, rectangular intersections, and T-shaped intersections. These scenarios were complex and varied, involving multiple intelligent agents such as pedestrians, cyclists, and vehicles, demonstrating rich and diverse interactive information.

[0042] It should be noted that the information obtained in this embodiment is all legal, does not involve privacy, and is information that is permitted to be obtained.

[0043] The acquired scene video is preprocessed by dividing it into frames, cropping them to a size of 1024×1024, and downsampling to 10Hz. This embodiment is geared towards trajectory prediction, therefore it is necessary to extract the agent's two-dimensional coordinate position in each frame from the preprocessed video recording to obtain trajectory information.

[0044] To detect moving agents in video frames, Faster-RCNN was used to perform target detection on video sequences containing motion information of agents such as pedestrians, cyclists, and vehicles. Bounding boxes were used to estimate the position and orientation of each agent. Subsequently, the target detection results were used as input to the DeepSort network, employing a Kalman filter and the Hungarian algorithm to track multiple targets. This yielded a matching correspondence between the agent, the current frame, and the agent's two-dimensional coordinates, thus obtaining the motion trajectories of each agent in the existing scene video and constructing a dataset.

[0045] Thus, a trajectory prediction dataset based on drone aerial photography of intersections, abbreviated as UITP dataset, has been constructed. The UITP dataset contains the following data fields: Frame_ID for each frame, Agent_ID for each agent, the two-dimensional coordinates (x and y) of the agent in each frame, and the semantic category of the agent. To facilitate model use of the data, the dataset is divided into training, validation, and test sets in a 70%, 10%, and 20% ratio, respectively.

[0046] Trajectory prediction can be viewed as a sequence prediction problem, that is, estimating the future trajectories of all agents in a scenario based on historical trajectories. Assume there are N interacting agents A1, A2, ..., A6 in the scenario. N Each intelligent agent A i Corresponding to a semantic category S c When S c When the value is 0, it means A i For vehicles; when S c When the value is 1, it means A i For pedestrians.

[0047] The motion trajectories of each agent in the dataset are divided into historical trajectories and future trajectories. Given agent A... i Historical time step T obs Observation trajectory sequence in The coordinates (in pixels) of the center point of the detected bounding box are represented as a historical trajectory sequence. The purpose of this embodiment is to generate a future time step T. pred Future trajectory sequence

[0048] The encoder constructs the scene as a dynamic spatiotemporal graph and extracts the trajectory distribution information of the agent from the dynamic spatiotemporal graph.

[0049] Abstract the current scene into a dynamic spatiotemporal graph G = (V, E) S E T ), where V is a set of nodes used to store the intelligent agents in the scene; E S It is a set of spatial edges used to connect two different agents at the same time step; ET This is a set of time edges used to connect agents of the same type at adjacent time steps. Nodes represent the agent's own motion information, while edges represent the social interaction information between agents. To represent agent A... i With A j Whether there is an interaction relationship between them is determined by using the l2 distance as a metric, and the calculation formula is shown in (1):

[0050]

[0051] in For agent A i A j Two-dimensional pixel coordinates, It is a semantic category S c The attention radius. When formula (1) holds, it means that agent A... i With A j Since they influence each other, it is believed that the edges between two agents are connected, meaning that they have a social interaction relationship.

[0052] During model training, a historical dynamic spatiotemporal graph is constructed based on historical trajectories.

[0053] The information extraction module extracts historical trajectory information, future trajectory information, and historical social interaction information between various intelligent agents from the historical dynamic spatiotemporal map and future trajectory, respectively. The historical trajectory information and historical social interaction information are fused to obtain historical fused information. The fused information and future trajectory information are then fused to obtain intelligent agent information.

[0054] The information extraction module includes a historical trajectory encoder, a social interaction encoder, a future trajectory encoder, and an information fusion module.

[0055] To make the predicted trajectory more accurate and feasible, this embodiment introduces a multimodal information encoder, which encodes the agent's historical trajectory information, social interaction information, and future trajectory information of the self-aware agent, and then fuses the various types of information to provide sufficient guidance for trajectory prediction.

[0056] ① Encoding of agent's historical trajectory

[0057] First, the agent's position coordinates The feature vectors are embedded into a fixed dimension, as shown in Equation (2). Here, φ(.) represents the embedding function with ReLU nonlinearity, and W... e Indicates the embedding weight. This represents the obtained fixed-dimensional feature vector.

[0058]

[0059] Then, in order to extract joint features from the historical trajectories of all agents, the obtained feature vectors are input into a Long Short-Term Memory (A-LSTM) network with 128 hidden dimensions to obtain historical trajectory information, expressed by the formula:

[0060]

[0061] Where A-LSTM represents the agent's historical trajectory encoder, W represents the hidden state vector of the A-LSTM network at time t. a These represent the weights of the A-LSTM network. These parameters are shared among all agents in the scene.

[0062] ② Social interaction coding

[0063] The movement of surrounding agents influences the decision-making of the target agent, thus requiring modeling of the social interaction effects between agents. This embodiment aggregates edges in the historical dynamic spatiotemporal graph based on semantic categories. By summing the states of all adjacent nodes of a specific type of edge and inputting the results into an encoder for combination, the social interaction information of each agent is obtained, expressed by the formula:

[0064]

[0065]

[0066] in, This represents the two-dimensional pixel coordinates of agent i at time t. This indicates that agent i follows the semantic category S. c The neighborhood group, This indicates that agent i moves along semantic category S at time t. c The edge aggregation vector, I-LSTM represents a Long Short-Term Memory network with 128 hidden dimensions. W represents the hidden state vector of the I-LSTM network at time t. i These represent the weights of the I-LSTM network. These parameters are shared among all edges of the same type.

[0067] By concatenating the historical encoded information of nodes with the social interaction information of edges, the hidden state of a single node can be generated. Historical fusion information is obtained, as shown in formula (6), where This indicates a concat connection.

[0068]

[0069] ③ Encoding the future trajectory of the intelligent agent

[0070] During the model training phase, the agent's true future trajectory Y is encoded using a bidirectional F-LSTM network with 32 hidden dimensions to obtain the hidden states. The future trajectory information is obtained, which can be expressed by the formula:

[0071]

[0072] Where Y represents the agent's true future trajectory at the prediction time, and F-LSTM represents a bidirectional long short-term memory network with 32 hidden dimensions. W represents the hidden state vector of the F-LSTM network at time t. y This represents the weights of the F-LSTM network.

[0073] During the model testing phase, since the agent's true future trajectory Y is unavailable, there is no need to encode it.

[0074] The encoder using parts ①②③ encodes multiple types of information. The information fusion module fuses the hidden states from the three parts to obtain the final vector representation. Information for the intelligent agent:

[0075]

[0076] The CVAE module is trained by fusing historical information and agent information to obtain a trained CVAE module, and the trajectory distribution information of the agent is output through the trained CVAE module.

[0077] The CVAE module includes a prior network, a recognition network, and a generative network. Agent information is input into the recognition network to obtain the distribution mean. Covariance Historical fusion information is input into the prior network to obtain the distribution mean. Covariance And calculate the distribution mean. Covariance Gaussian hidden variable Zq and distribution mean Covariance Gaussian hidden variable Z p Through Gaussian hidden variable Z q and Gaussian hidden variable Z p The KL divergence loss is calculated to train the CVAE module. After training, the trained CVAE module is obtained, and the Gaussian latent variable Z is then processed through a generative network. q It is linked with the agent's information to output the agent's trajectory distribution information.

[0078] Specifically, the movement of an intelligent agent is multimodal; there are often multiple reasonable and feasible trajectories for moving from starting point A to ending point B. To learn and predict the distribution P(Y|X) of future trajectories, this embodiment employs a conditional variational autoencoder (CVAE) module, introducing Gaussian latent variables Z~N(μ) Z ,σ Z This explains multimodality.

[0079] The CVAE module consists of three parts: the prior network P θ (Z|X), Identifying Network Q φ (Z|X,Y) and generator network P ψ (Y|X,Z), which are implemented through fully connected layers, where θ, φ, and ψ represent the parameters of the three networks. Equation (9) illustrates the implementation process of the prior network. The implementation methods of the recognition network and the generation network are similar to those in Equation (9), and will not be elaborated here.

[0080]

[0081] in, FC represents the output of formula (8). θ (·) indicates a fully connected layer network. Indicates FC θ The weight of (·).

[0082] During the training phase, the encoder's final state vector That is, the agent information is input into the recognition network Q. φ Predicted distribution mean in (Z|X,Y) Covariance This is to capture the dependency between the observed trajectory and the true trajectory. Since the prior network does not know the true trajectory Y beforehand, the encoder's final state vector is... that is Therefore, it is only necessary to integrate historical information. As a priori network P θ The input (Z|X) is used to predict the distribution mean. Covariance Through optimization and The KL divergence loss is used to learn the dependency between Y and X from the prior network. The latent variable Z is obtained from sampling. q and then combine it with the encoder's final state vector. That is, the feature vector is obtained by concatenating the information of the intelligent agent. Expressed as a formula:

[0083]

[0084] During the testing phase, from Mid-sampling yields latent variable Z p Similar to formula (10), the generator network will use the Gaussian hidden variable Z p The feature vector is obtained by concatenating and fusing information. Whether in the training or testing phase, vectors It will affect the subsequent endpoint driver module and decoder.

[0085] The movement of an intelligent agent exhibits significant randomness and uncertainty. Therefore, this embodiment introduces an endpoint-driven module. This module learns the agent's final movement intention before predicting the complete trajectory, thus providing guidance for trajectory prediction. The endpoint of the agent's true future trajectory is defined. This value is known during the training phase but unknown during the testing phase. Let... Indicates time step T obs +T pred The parameters of the Gaussian mixture model have K components, where each Gaussian component can be viewed as a distribution of trajectory modes, expressed by the formula:

[0086]

[0087] Among them, each π j This represents the probability that the agent's movement belongs to this mode, and their sum is 1. This indicates that agent i is at time step T. obs +T pred The mean and covariance of the corresponding j-th Gaussian component.

[0088] The endpoint-driven EDModule proposed in this embodiment does not directly predict trajectory coordinates, but rather predicts the parameters of the trajectory distribution and calculates the predicted future trajectory endpoint. The process is shown in the following formula:

[0089]

[0090] in, and Let represent the mean, variance, and correlation of the j-th Gaussian component corresponding to agent i, respectively, and MLP(·) denotes the linear embedding function.

[0091]

[0092] in, and Let GMM(·) represent the mean, variance, and correlation of the j-th Gaussian component corresponding to the predicted endpoint of agent i, respectively. GMM(·) represents a Gaussian mixture model.

[0093]

[0094] in, Indicates from satisfying and The predicted endpoint is obtained by sampling from a Gaussian mixture model with K components.

[0095] As prediction time increases, the information carried in the decoder gradually weakens, thus affecting the accuracy of long-term prediction. To address this issue, this embodiment proposes a decoder based on inverse enhancement decoding. The decoder includes a forward decoder and an inverse decoder. By fusing the inverse decoder with the forward decoder, the role of the hidden state vector is enhanced.

[0096] vector With the predicted future trajectory endpoint Connect the two lines as input to the forward decoder. And initialize the hidden state to As shown in formula (16):

[0097]

[0098]

[0099] F-GRU stands for forward decoder. This represents the input to the F-GRU network. W represents the hidden state of the F-GRU at time t. f This represents the weight of an F-GRU unit; the weight parameters are shared throughout the network.

[0100] Over time, the state variables obtained through the encoder The effect of vector propagation gradually weakens, leading to increasingly larger prediction errors with longer prediction step sizes. Therefore, this paper introduces a reverse decoding approach, which transforms the vector... As the input to the inverse decoder, it is shown in Equation (17):

[0101]

[0102] B-GRU stands for reverse decoder. This indicates that the B-GRU operates at time t+T pred The hidden state of W b This represents the weight of a B-GRU unit; the weight parameters are shared throughout the network.

[0103] After decoding using a forward decoder and a backward decoder, at each time step, the backward hidden state vector is concatenated with the forward hidden state vector, thereby enhancing the role of the hidden state vector at each time step. The concatenated hidden vector is then used to predict the trajectory point coordinates at that time step, expressed by formula (18):

[0104]

[0105] in, This indicates that at time t+T pred -1 The hidden state vector of the forward decoder. This indicates that at time t+T pred -1 The hidden state vector of the inverse decoder, W of W ob and b o These represent the weights and biases of the fully connected layer at output. This indicates that at time t+T pred -1 is the coordinate of the predicted trajectory point.

[0106] The UITP dataset has a sampling frequency of 10Hz. A 3-second historical trajectory is selected. To predict the future trajectory after 3 seconds, the parameter T is set... obs =30, T pred =30. An LSTM network is used as the backbone network for the encoder, where the dimensions of A-LSTM are 128, I-LSTM are 128, and F-LSTM are 32; a GRU network is used as the backbone network for the decoder, where the dimensions of F-GRU and B-GRU are 256. In formula (1), the dimension of the embedding layer is 32, the dimension of the prior network in the CVAE module is 32, and the dimension of the latent variable Z is 25.

[0107] This embodiment also proposes a new loss function L. EDRENet It consists of three parts: the evidence lower bound (ELBO) for negative log-likelihood estimation, the negative log-likelihood loss for endpoint prediction, and the Kullback-Leibler Divergence loss used to train the conditional variational autoencoder. The corresponding calculation formulas are as follows:

[0108]

[0109]

[0110] L3(φ,θ)=D KL (Q φ (Z|X,Y)||P θ (Z|X)) (21)

[0111] Therefore, the embodiment uses multiple loss functions to optimize the parameters of the agent trajectory prediction network model. The overall loss function is shown in Equation (22), where α, β, and γ are set to 1.

[0112]

[0113] To avoid overfitting, this embodiment employs a suitable data preprocessing strategy, rotating all trajectories in the scene around the scene origin at 15° intervals between 0° and 360°. Using the backpropagation algorithm, the loss function (22) is used as the cost function to train the agent trajectory prediction network model proposed in this embodiment, obtaining a trained agent trajectory prediction network model. This model is then used to identify the acquired current scene video and obtain the predicted future trajectory of the agent in the current scene. During training, the batch size is set to 256, and the epoch size is set to 150.

[0114] The trained agent trajectory prediction network identifies the current scene video and obtains the predicted future trajectory of the agent in the current scene as follows:

[0115] Determine the motion trajectories of each agent in the current scene video;

[0116] Based on the motion trajectories of each agent, a dynamic spatiotemporal graph of the agents in the current scene is constructed. The set of nodes in the dynamic spatiotemporal graph is used to store the agents in the current scene, the set of spatial edges is used to connect two different agents at the same time step, and the set of temporal edges is used to connect the same agent at adjacent time steps.

[0117] The information extraction module extracts the motion trajectory information and social interaction information of each agent from the dynamic spatiotemporal graph, and then fuses the motion trajectory information and social interaction information to obtain fused information.

[0118] The fused information is identified by the trained CVAE module to obtain the trajectory distribution information of the agent;

[0119] The endpoint-driven module identifies the trajectory distribution information of the agent and obtains the predicted future trajectory endpoint of the agent.

[0120] Based on trajectory distribution information, predicted future trajectory endpoints, and a decoder, the agent's future trajectory is predicted to obtain a trajectory prediction result. The decoder includes a forward decoder and a backward decoder. The trajectory distribution information and the predicted future trajectory endpoints are concatenated and input into the forward decoder to obtain a forward hidden state. The trajectory distribution information is input into the backward decoder to obtain a backward hidden state. The forward hidden state and the backward hidden state are concatenated to predict the agent's future trajectory to obtain a trajectory prediction result.

[0121] The performance of the intelligent agent trajectory prediction network model disclosed in this embodiment is verified.

[0122] The hardware requirements for the verification experiment were: Ubuntu 16.04L, CPU: Intel Core i9-9900X, RAM: 64G, and 3 TITAN XP graphics cards.

[0123] The experimental data comes from the UITP dataset, captured by drones at several intersections in Lixia District, Jinan City. It contains rich information on the movement and interaction of intelligent agents such as pedestrians, cyclists, and vehicles. The movement trajectories of these agents were obtained using existing detection and tracking algorithms.

[0124] ADE, FDE, and KDE NLL were selected to evaluate the trajectory prediction performance of the model disclosed in this embodiment. The smaller the value of these three indicators, the more accurate the predicted future trajectory.

[0125] (1) Average Displacement Error (ADE): This is used to calculate the average Euclidean distance between the actual trajectory and the predicted trajectory within the predicted time step. It reflects the average prediction performance of the model. The calculation formula is shown in (23).

[0126]

[0127] In formula (23), and These represent the time intervals of agent i and agent i, respectively. obs +t represents the predicted trajectory and the actual future trajectory, where N represents the number of agents in the scenario, and T... pred Indicates the predicted duration.

[0128] (2) Final Displacement Error (FDE): This is used to calculate the average Euclidean distance between the actual trajectory and the predicted trajectory at the final position. It reflects the final prediction accuracy of the model. The calculation formula is shown in (24).

[0129]

[0130] In formula (24), and These represent the time intervals of agent i and agent i, respectively. obs +t represents the predicted trajectory and the actual future trajectory, where N represents the number of agents in the scenario, and T... obs and T pred These represent the observation duration and the prediction duration, respectively.

[0131] (3) Kernel density estimation based on negative log-likelihood KDE NLL: At each prediction time step, the probability density function of the sampled trajectory at that time step is obtained using kernel density estimation. Based on these estimates, the average negative log-likelihood of the true trajectory is calculated.

[0132] Experiments on agent trajectory prediction were conducted on the UITP dataset. Table 1 compares the disclosed method of this embodiment with existing methods, with smaller values ​​indicating better performance; Table 2 shows the relationship between prediction error and prediction time, and compares the disclosed method of this embodiment with the benchmark method Trajectron++; Table 3 presents the results of the ablation experiment, demonstrating the effectiveness of the endpoint-driven module and the decoder based on reverse enhancement decoding.

[0133] In addition to the quantitative results, this embodiment also performs a qualitative analysis of the trajectory prediction results. For example... Figure 3 As shown in the figure, the observed trajectory represents the agent's historical trajectory, the ground truth represents the agent's true future trajectory, and the predicted trajectories represent the agent's predicted trajectories. Twenty trajectories are selected as the distribution of future trajectories, and the observed trajectory, true future trajectory, and predicted trajectory of nodes 795, 799, and 793 are visualized. Figure 4 As shown in the figure, Observation represents the agent's historical trajectory, GroundTruth represents the agent's true future trajectory, Trajectron++ represents the prediction effect of the baseline method, and EDRENet represents the prediction effect of the network proposed in this invention. The performance of the EDRENet model proposed in this embodiment and the baseline method Trajectron++ in multimodal prediction is compared. It can be seen that the multiple trajectories generated by the method disclosed in this embodiment are more concentrated and clustered, and the true trajectory is always distributed in the darker, higher-density region.

[0134] Table 1 Prediction Results of Experimental Data

[0135]

[0136]

[0137] Table 2 Relationship between prediction error and prediction time

[0138]

[0139] Table 3 Ablation Experiment Results

[0140] BaseLine ED Module RE Decoder ADE FDE √ 5.82 10.31 √ √ 4.84 9.69 √ √ 4.86 9.70 √ √ √ 4.78 9.63

[0141] This embodiment discloses a method that proposes an agent trajectory prediction network (EDRENet) model based on endpoint-driven and reverse-enhanced decoding. First, the scene is constructed as a dynamic spatiotemporal graph to effectively model the social interactions between agents. Second, a multimodal information encoder is introduced to encode the historical trajectories of agents, the interaction information between agents, and the future trajectories of agents, extracting various state information. Third, a CVAE module is used to explain multimodality by introducing latent variables. Then, an endpoint-driven module is used to predict the endpoint before predicting the complete trajectory, thus providing guidance for trajectory prediction. Finally, a decoder based on reverse-enhanced decoding is proposed to fuse positive and negative hidden state vectors to obtain diverse trajectories that conform to physical and social acceptability rules, improving the problem of inaccurate long-term prediction. Results show that the method proposed in this embodiment effectively solves the problem of multi-object trajectory prediction in complex traffic scenarios, and addresses the problems of insufficient modeling of agent interaction information and poor long-term prediction performance in existing technologies in complex traffic scenarios, significantly improving the accuracy of trajectory prediction.

[0142] Example 2

[0143] In this embodiment, an intelligent agent trajectory prediction system is disclosed, comprising:

[0144] The dynamic spatiotemporal graph acquisition module is used to acquire the current scene video and construct the dynamic spatiotemporal graph of the agent in the current scene;

[0145] The trajectory distribution information acquisition module is used to identify dynamic spatiotemporal graphs and obtain the trajectory distribution information of the intelligent agent;

[0146] The future trajectory endpoint prediction module is used to identify the trajectory distribution information of the agent through the endpoint driving module and obtain the predicted future trajectory endpoint of the agent.

[0147] The future trajectory prediction module is used to predict the future trajectory of the agent based on trajectory distribution information, the predicted future trajectory endpoint, and the decoder, and obtain the trajectory prediction result. The decoder includes a forward decoder and a backward decoder. The trajectory distribution information and the predicted future trajectory endpoint are connected and then input into the forward decoder to obtain the forward hidden state. The trajectory distribution information is input into the backward decoder to obtain the backward hidden state. The forward hidden state and the backward hidden state are connected to predict the future trajectory of the agent and obtain the trajectory prediction result.

[0148] Example 3

[0149] In this embodiment, an electronic device is disclosed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps described in the intelligent agent trajectory prediction method disclosed in Embodiment 1.

[0150] Example 4

[0151] In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions, which, when executed by a processor, complete the steps described in the intelligent agent trajectory prediction method disclosed in Embodiment 1.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for predicting trajectories of agents, the method comprising: The method comprises the following steps: acquiring a current scene video, and constructing a dynamic space-time graph of agents in the current scene; extracting motion trajectory information of the agents and social interaction information of each agent from the dynamic space-time graph through an information extraction module, fusing the motion trajectory information and the social interaction information, and obtaining fused information; identifying the fused information through a trained CVAE module, and obtaining trajectory distribution information of the agents; identifying the trajectory distribution information of the agents through an end point driving module, and obtaining a predicted future trajectory end point of the agents; predicting a future trajectory of the agents according to the trajectory distribution information, the predicted future trajectory end point and a decoder, and obtaining a trajectory prediction result, wherein the decoder comprises a forward decoder and a backward decoder, the trajectory distribution information and the predicted future trajectory end point are connected and input into the forward decoder to obtain a forward hidden state, the trajectory distribution information is input into the backward decoder to obtain a backward hidden state, and the forward hidden state and the backward hidden state are connected to predict the future trajectory of the agents and obtain the trajectory prediction result.

2. The method of claim 1, wherein, determining motion trajectories of agents in a current scene video; constructing a dynamic space-time graph of agents in the current scene according to the motion trajectories of the agents, wherein a node set of the dynamic space-time graph is used to store the agents in the current scene, a space edge set is used to connect two different agents at a same time step, and a time edge set is used to connect a same agent at adjacent time steps.

3. The method of claim 1, wherein, acquiring motion trajectories of agents in an existing scene video, and constructing a dataset; dividing the motion trajectories of the agents in the dataset into historical trajectories and future trajectories; constructing a historical dynamic space-time graph according to the historical trajectories; extracting historical trajectory information, future trajectory information and historical social interaction information of the agents from the historical dynamic space-time graph and the future trajectories respectively through an information extraction module, fusing the historical trajectory information and the historical social interaction information to obtain historical fused information, and fusing the historical fused information and the future trajectory information to obtain agent information; training a CVAE module through the historical fused information and the agent information, and obtaining a trained CVAE module.

4. The method of claim 3, wherein, The CVAE module includes a prior network, a recognition network, and a generative network. Agent information is input into the recognition network to obtain the distribution mean. Covariance By inputting historical fusion information into the prior network, the distribution mean is obtained. Covariance Calculate the distribution mean Covariance Gaussian hidden variable Z q and distribution mean Covariance Gaussian hidden variable Z p Through Gaussian hidden variable Z q and Gaussian hidden variable Z p Calculate the KL divergence loss, train the CVAE module, obtain the trained CVAE module, and then use a generative network to process the Gaussian latent variable Z. q It is linked with the agent's information to output the agent's trajectory distribution information; The fusion information is input into the prior network of the trained CVAE module to obtain distribution mean and covariance , and the Gaussian latent variable Z of the distribution mean and covariance p is calculated p The generation network concatenates the Gaussian latent variable Z and the fusion information to obtain the trajectory distribution information of the agent.

5. The method of claim 1, wherein, The end-point driving module calculates the predicted future trajectory end-point The formula is: ， wherein, represents the trajectory distribution information of the agent, represents the weight, , and respectively represent the mean, variance and correlation of the corresponding first Gaussian component of the agent , represents the linear embedding function; , and respectively represent the mean, variance and correlation of the corresponding first Gaussian component of the predicted end point of the agent , represents the Gaussian mixture model; represents the predicted end point sampled from the Gaussian mixture model with , and components satisfying .

6. The method of claim 1, wherein, The forward decoder adopts a trained F-GRU model, and the backward decoder adopts a trained B-GRU model.

7. An agent trajectory prediction system, comprising: The method comprises the following steps: a dynamic space-time graph acquisition module is used to acquire a current scene video, and construct a dynamic space-time graph of agents in the current scene; a trajectory distribution information acquisition module is used to extract motion trajectory information of the agents and social interaction information of each agent from the dynamic space-time graph through an information extraction module, fuse the motion trajectory information and the social interaction information, and obtain fused information; and identify the fused information through a trained CVAE module, and obtain trajectory distribution information of the agents; a future trajectory end point prediction module is used to identify the trajectory distribution information of the agents through an end point driving module, and obtain a predicted future trajectory end point of the agents; A future trajectory prediction module is configured to predict a future trajectory of the agent according to the trajectory distribution information, the predicted future trajectory endpoint, and a decoder to obtain a trajectory prediction result, wherein the decoder comprises a forward decoder and a backward decoder, the trajectory distribution information and the predicted future trajectory endpoint are connected and input into the forward decoder to obtain a forward hidden state, the trajectory distribution information is input into the backward decoder to obtain a backward hidden state, the forward hidden state and the backward hidden state are connected to predict the future trajectory of the agent to obtain the trajectory prediction result.

8. An electronic device, comprising: A computer program product comprising a memory and a processor, and computer instructions stored in the memory and run on the processor, when the computer instructions are run by the processor, the steps of the agent trajectory prediction method in any one of claims 1-6 are completed.

9. A computer-readable storage medium, characterized in that, A computer program product for storing computer instructions, when the computer instructions are executed by a processor, the steps of the agent trajectory prediction method in any one of claims 1-6 are completed.

Images