An agent trajectory navigation system based on visual language and a method thereof

Abstract

The application discloses a kind of based on visual language to agent trajectory navigation system and method thereof, navigation system includes history trajectory processing module, trajectory time series diagram model, dynamic trajectory interaction model, first trajectory prediction module, second trajectory prediction module and optimization trajectory prediction module;The dynamic interaction model introduces the edge weight according to the distance relationship change between agent between graph adjacent node update and obtains the dynamic interaction relationship of agent;The first trajectory prediction module is based on target body dynamic interaction relationship to graph convolution network training and obtains the dynamic interaction feature sequence of agent;The second prediction module is based on attention mechanism to agent dynamic interaction feature sequence training and obtains the motion trend trajectory sequence of agent;The optimization trajectory prediction module is based on diversity perception loss function to agent motion trend trajectory sequence calculation output optimal agent motion trajectory, the present application assists agent to provide guarantee in the safe navigation of complex environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent transportation technology, and in particular to a visual language-based intelligent agent trajectory navigation system and method, applicable to application scenarios such as autonomous driving, intelligent robots, and intelligent monitoring that require modeling and predicting the future movement trends of intelligent agents in complex dynamic scenes. Background Technology

[0002] Trajectory prediction for intelligent agents is a key issue for achieving safe and efficient navigation in autonomous driving systems and intelligent robots. In dynamic and complex traffic or public environments, the movement of intelligent agents is not only driven by their own intentions but also profoundly influenced by environmental interactions with surrounding people and environmental constraints. Therefore, accurately predicting the future trajectory of intelligent agents in highly uncertain environments is a core challenge for ensuring the obstacle avoidance capabilities and decision-making safety of intelligent systems.

[0003] Existing trajectory prediction methods mainly fall into three categories: physical model methods, machine learning methods, and deep learning methods. (1) Physical model methods are based on classical mechanics principles and deduce trajectories by establishing an interactive mechanical model between the agent and the environment, such as environmental force models. These methods can intuitively reflect the laws of motion, but they rely on a large number of prior assumptions, have insufficient adaptability, and are difficult to promote and apply in complex scenarios. (2) Machine learning methods predict trajectories by performing feature engineering on historical trajectories and using traditional classification and regression models (such as Gaussian processes, decision trees, Bayesian networks, etc.). These methods have low computational cost and high flexibility, but they rely heavily on feature engineering and are prone to overfitting when dealing with high-dimensional complex interactions, resulting in weak interpretability. (3) Deep learning methods have shown outstanding performance in recent years. Models based on recurrent neural networks, graph convolutional networks, and Transformers have significant advantages in high-dimensional spatiotemporal data modeling. They can automatically extract spatiotemporal features from trajectory data and improve prediction accuracy and generalization to a certain extent. However, existing deep learning methods still have two limitations: First, they lack physical rationality and interpretability, usually modeling environmental interactions as black box processes and ignoring kinematic differences; second, they are difficult to balance prediction accuracy and diversity, as overemphasizing accuracy can lead to prediction pattern collapse, while overemphasizing diversity can reduce the reliability of predicted trajectories.

[0004] In summary, existing methods struggle to maintain the physical plausibility of predictions while simultaneously ensuring the accuracy and diversity of prediction results. Therefore, a new technical solution is urgently needed to introduce physical constraints into graph structure modeling and achieve multimodal trajectory prediction through an improved loss function, thereby enhancing the interpretability, robustness, and practical application value of the method. Summary of the Invention

[0005] To address the shortcomings of existing intelligent agent trajectory prediction methods: on the one hand, existing methods mostly model environmental interactions as black-box processes, lacking physical rationality and interpretability; on the other hand, it is difficult to achieve a balance between accuracy and diversity in prediction results, which can easily lead to pattern collapse or prediction bias, thereby reducing the robustness and reliability of the system in complex dynamic environments. This invention proposes an intelligent agent trajectory prediction method and system based on physical constraint graph convolutional networks and diversity perception loss optimization.

[0006] To address the problems existing in the prior art, the present invention adopts the following technical solution:

[0007] A visual language-based trajectory navigation system for intelligent agents includes a historical trajectory processing module, a trajectory time sequence graph model, a dynamic trajectory interaction model, a first trajectory prediction module, a second trajectory prediction module, and an optimized trajectory prediction module; wherein:

[0008] The historical trajectory processing module collects the position and velocity of the agent in the scene to obtain the historical trajectory sequence of the agent.

[0009] The trajectory temporal interaction graph model constructs a temporal interaction graph between agents at each time step based on the historical trajectory sequence.

[0010] The dynamic interaction model introduces edge weights that update adjacent nodes in the graph based on changes in distance relationships between agents to obtain dynamic interaction relationships between agents.

[0011] The first trajectory prediction module obtains the dynamic interaction feature sequence of the agent by training a graph convolutional network based on the dynamic interaction relationship of the target;

[0012] The second prediction module obtains the agent's motion trend trajectory sequence by training the dynamic interaction feature sequence of the agent based on the attention mechanism;

[0013] The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the diversity perception loss function for the agent's motion trend trajectory sequence.

[0014] Furthermore, the trajectory temporal interaction graph model The nodes of the trajectory temporal interaction graph are agents, and the edge set of the trajectory temporal interaction graph represents the distance relationships between agents. The initial features of the nodes include a concatenated vector of position and velocity, and the velocity is estimated by the position difference between adjacent time steps to enhance the representation ability of local motion trends.

[0015] node Represents intelligent agents At any moment State characteristics, including location and speed ,Right now:

[0016] ;

[0017] Edge set Based on distance threshold Build: When At that time, node With nodes Connected.

[0018] Furthermore, the edge weights are:

[0019] ;

[0020] in: Let be the Euclidean distance between the rows. These are learnable parameters used to control the interaction strength, decay rate, and range of influence.

[0021] Furthermore, the first trajectory prediction module is:

[0022] ;

[0023] in For nodes The neighborhood group, and For the first Layer parameters, This is the activation function.

[0024] Furthermore, the diversity-aware loss function is:

[0025] ;

[0026] in: This is the mean squared error loss, used to ensure prediction accuracy; To mitigate diversity loss, the diversity of candidate trajectories is encouraged; To maintain constant values, avoid numerical instability, and constrain diversity. For diversity constraints, that is, to adopt any of the following implementations:

[0027] (i) Based on the variety mechanism of “best matching selection”, supervision is only performed on the candidate with the smallest error with the real trajectory;

[0028] (ii) Based on the diversity regularization term of the similarity suppression between candidate trajectories, cosine similarity / kernel similarity is used to punish excessive similarity between candidates under the same input in order to avoid pattern collapse.

[0029] Furthermore, the attention mechanism is a Transformer structure, which includes a multi-head attention layer, a feedforward sub-layer, residual connections and layer normalization, and applies position / time step encoding to temporal features to capture long-term dependencies.

[0030] The present invention can also adopt the following scheme: a method for navigating the trajectory of an intelligent agent based on visual language.

[0031] S1. Obtain the historical trajectory sequence of at least one agent in the target scene. ,in Represents two-dimensional coordinates;

[0032] S2. Based on the historical trajectory at each time step Constructing a time-series interaction graph intelligent agents Represented as a trajectory time-series interaction graph node Its state characteristics include position With speed And based on the distance between intelligent rows, it forms the edge set of the temporal interaction graph. ;

[0033] S3. For any pair of adjacent nodes in the graph Calculate edge weights The node representations are updated in the graph convolutional network using the message passing rules modulated by the edge weights to obtain the dynamic interaction relationships of the agents; wherein: the edge weights The following formula is given:

[0034] ;

[0035] in: Let be the Euclidean distance between the rows. These are learnable parameters used to control interaction strength, decay rate, and range of influence; the weight normalization employs neighborhood-based row normalization or softmax normalization to ensure that the weights of each neighboring edge are numerically stable and comparable; the prediction interval length... Observation interval length and generate at least for each input sample Candidate future trajectories are selected to enhance multimodal coverage;

[0036] S4. Based on the dynamic interaction relationship of the target, train the graph convolutional network to obtain the dynamic interaction feature sequence of the agent;

[0037] S5. Obtain the agent's motion trend trajectory sequence by training the dynamic interaction feature sequence of the agent based on the attention mechanism. ;

[0038] S6. The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the diversity-aware loss function for the agent's motion trend trajectory sequence; it jointly trains the parameters of the graph convolutional network and the time series prediction module based on a stochastic gradient descent-type optimizer, iterating in batches until convergence, and optionally employing an early stopping strategy to improve generalization performance, wherein: the diversity-aware loss is:

[0039] .

[0040] Beneficial effects

[0041] 1. Enhance physical rationality and interpretability: By introducing physically inspired edge weights into graph convolutional networks, the interaction modeling between agents conforms to dynamic intuition and can explain motion behavior under different density environments.

[0042] 2. Balancing accuracy and diversity: By designing diversity-perceived loss, the mean squared error is combined with diversity constraints, and a logarithmic regularization term is introduced to achieve a dynamic balance between accuracy and coverage in the prediction results, thus avoiding model collapse.

[0043] 3. Improve robustness and generalization: By combining graph convolution features with attention mechanisms, the method performs stably in both dense and sparse scenes, enhancing its adaptability to complex dynamic environments.

[0044] 4. Wide range of applications: This invention can output multimodal and physically reasonable predicted trajectories, which can be directly applied to path planning of autonomous vehicles, obstacle avoidance of intelligent robots, and risk warning of intelligent monitoring systems, providing reliable support for safe and efficient navigation in complex environments. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating the definition of the intelligent agent trajectory prediction problem according to an embodiment of the present invention, showing the modeling methods for historical trajectories, future trajectories, and environmental interactions;

[0046] Figure 2 This invention is a schematic diagram of the structure of an intelligent agent navigation system based on visual language;

[0047] Figure 3 This invention relates to spatiotemporal graph construction, edge weight calculation, and message passing process based on mechanical constraints.

[0048] Figure 4 This is a comparative diagram of different loss functions involved in the present invention, showing the differences in prediction results based on L2 loss, variety loss and diversity-perceived loss respectively;

[0049] Figure 5This is a visualization of the prediction results of the embodiments of the present invention on typical datasets (ETH, HOTEL, UNIV, ZARA01, ZARA02), which compares the trajectory prediction performance of the method of the present invention with that of existing models in different scenarios. Detailed Implementation

[0050] The following is in conjunction with the appendix Figure 1-5 The present invention will be described in detail below:

[0051] This invention provides a visual language-based trajectory navigation system and method for intelligent agents. The invention mainly focuses on a trajectory prediction method for intelligent agents based on physically constrained graph convolutional networks and diversity-aware loss. This method introduces physically inspired edge weights during graph convolutional modeling to achieve dynamic modeling of interactions between target agents. At the same time, it proposes a diversity-aware loss function to jointly optimize prediction accuracy and trajectory diversity, thereby improving the physical rationality, interpretability, and multimodal coverage of the prediction results, effectively solving the limitations of existing technologies.

[0052] The navigation system of this invention includes a historical trajectory processing module, a trajectory time sequence graph model, a dynamic trajectory interaction model, a first trajectory prediction module, a second trajectory prediction module, and an optimized trajectory prediction module, such as... Figure 2 As shown; where:

[0053] The historical trajectory processing module collects the position and velocity of the agent in the scene to obtain the historical trajectory sequence of the agent.

[0054] Example 1

[0055] like Figure 1 As shown, this invention uses the historical trajectory of the intelligent agent as input. Let the target intelligent agent at time... The position is The trajectory within the observation interval is as follows:

[0056]

[0057] Predict the trajectory for the future interval:

[0058]

[0059] in To predict the step size, the prediction function can be formalized as:

[0060]

[0061] in Indicates environmental constraints. Indicates environmental interaction information, These are the model parameters.

[0062] The trajectory temporal interaction graph model constructs a temporal interaction graph between agents at each time step based on the historical trajectory sequence.

[0063] Example 2:

[0064] like Figure 3 As shown, this invention constructs an interaction graph between agents at each time step, using a combination of temporal interaction graph construction and physical constraint weighting. .

[0065] node Represents intelligent agents At any moment State characteristics, including location and speed ,Right now:

[0066]

[0067] Edge set Based on distance threshold Build: When At that time, node With nodes Connected. Among them:

[0068] (i) Based on radius threshold Adjacency relationship: when Connect edges at the same time; or

[0069] (ii) Based on - Adjacency relationship of nearest neighbors: Select the nearest neighbor for each node. The neighbors are connected, among which .

[0070] The dynamic interaction model introduces edge weights based on changes in distance relationships between agents to update adjacent nodes in the graph, thereby obtaining dynamic interaction relationships between agents. Building upon this, the invention introduces a physically inspired edge weight calculation method.

[0071]

[0072] in: ,parameter This is a learnable variable used to adjust the interaction strength, decay rate, and range of influence. The design simulates the repulsive force effect in an environmental force model, giving the edge weights physical validity.

[0073] The first trajectory prediction module obtains the dynamic interaction feature sequence of the agent by training a graph convolutional network based on the dynamic interaction relationship of the target;

[0074] Example 3:

[0075] After the interaction graph is constructed, graph convolutional feature extraction is performed using a physically constrained graph convolutional network (PhysGCN). Its update rule is as follows:

[0076]

[0077] in For nodes The neighborhood group, and For the first Layer parameters, This is the activation function. This process, through weighted aggregation of neighbor information, enables the modeling of dynamic interactions between agents.

[0078] The second prediction module obtains the agent's motion trend trajectory sequence by training the dynamic interaction feature sequence of the agent based on the attention mechanism; Example 4: Time series prediction modeling as follows Figure 2 As shown, the interaction features extracted by PhysGCN are input into the time series prediction module. This module adopts a Transformer structure based on an attention mechanism, including:

[0079] A multi-head attention layer is used to capture global dependencies in trajectory data;

[0080] Feedforward sublayers and layer normalization units are used to stabilize training and enhance nonlinear modeling capabilities. Through multi-layer stacking, the model can effectively learn motion trends over long periods and generate candidate sequences of future trajectories.

[0081] The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the diversity perception loss function for the agent's motion trend trajectory sequence.

[0082] Example 5:

[0083] like Figure 4 As shown, the diversity-aware loss is optimized to balance prediction accuracy and diversity. This invention designs a diversity-aware loss function:

[0084] ;

[0085] in:

[0086] This is the mean squared error loss, used to ensure prediction accuracy;

[0087] To mitigate diversity loss, the diversity of candidate trajectories is encouraged;

[0088] It is a constant to avoid numerical instability.

[0089] This loss avoids the collapse of a single prediction pattern while ensuring the physical plausibility of the trajectory.

[0090] This invention also employs a visual language-based method for navigating the trajectory of an intelligent agent, comprising the following steps:

[0091] S1. Obtain the historical trajectory sequence of at least one agent in the target scene. ,in Represents two-dimensional coordinates;

[0092] S2. Based on the historical trajectory at each time step Constructing a time-series interaction graph intelligent agents Represented as a trajectory time-series interaction graph node Its state characteristics include position With speed And based on the distance between intelligent rows, it forms the edge set of the temporal interaction graph. ;

[0093] S3. For any pair of adjacent nodes in the graph Calculate edge weights The node representations are updated in the graph convolutional network using the message passing rules modulated by the edge weights to obtain the dynamic interaction relationships of the agents; wherein: the edge weights The following formula is given:

[0094] ;

[0095] in: Let be the Euclidean distance between the rows. These are learnable parameters used to control the interaction strength, decay rate, and range of influence.

[0096] S4. Based on the dynamic interaction relationship of the target, train the graph convolutional network to obtain the dynamic interaction feature sequence of the agent;

[0097] S5. Obtain the agent's motion trend trajectory sequence by training the dynamic interaction feature sequence of the agent based on the attention mechanism. ;

[0098] S6. The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the agent's motion trend trajectory sequence using a diversity perception loss function; wherein: the diversity perception loss is:

[0099] .

[0100] Although the present invention has been described above, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit of the present invention, and these modifications are all within the protection scope of the present invention.

Claims

1. A visual language-based intelligent agent trajectory navigation system, characterized in that, The navigation system includes a historical trajectory processing module, a trajectory time-series interactive graph model, a dynamic trajectory interactive model, a first trajectory prediction module, a second trajectory prediction module, and an optimized trajectory prediction module; among which: The historical trajectory processing module collects the position and velocity of the agent in the scene to obtain the historical trajectory sequence of the agent. The trajectory temporal interaction graph model constructs a temporal interaction graph between agents at each time step based on the historical trajectory sequence; wherein: The trajectory time-series interaction graph model The nodes of the trajectory temporal interaction graph are agents, and the edge set of the trajectory temporal interaction graph represents the distance relationships between agents; wherein: the initial features of the nodes include a concatenated vector of position and velocity, and the velocity is estimated by the position difference between adjacent time moments to enhance the representation ability of local motion trends. Represents intelligent agents At any moment State characteristics, including location and speed ,Right now: ； The edge set Based on distance threshold Build: When At that time, node With nodes Connected; The dynamic interaction model introduces edge weights based on changes in distance relationships between agents to update adjacent nodes in the graph, thereby obtaining dynamic interaction relationships between agents; wherein: the edge weights are: ； in: Let be the Euclidean distance between the rows. These are learnable parameters used to control the interaction strength, decay rate, and range of influence. The first trajectory prediction module obtains the dynamic interaction feature sequence of the agent by training a graph convolutional network based on the dynamic interaction relationship of the target; The second trajectory prediction module obtains the agent's motion trend trajectory sequence by training the agent's dynamic interaction feature sequence based on the attention mechanism; The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the agent's motion trend trajectory sequence using a diversity-aware loss function, wherein: the diversity-aware loss function is: ； in: This is the mean squared error loss, used to ensure prediction accuracy; To mitigate diversity loss, the diversity of candidate trajectories is encouraged; To maintain constant values, avoid numerical instability, and constrain diversity. To constrain diversity, it can be implemented in any of the following ways: (i) Based on the variety mechanism of "best matching selection", supervision is only performed on the candidate with the smallest error with the real trajectory; (ii) Based on the diversity regularization term of the similarity suppression between candidate trajectories, cosine similarity / kernel similarity is used to punish excessive similarity between candidates under the same input in order to avoid pattern collapse.

2. The visual language-based intelligent agent trajectory navigation system according to claim 1, characterized in that, The first trajectory prediction module is: ； in For nodes The neighborhood group, and For the first Layer parameters, This is the activation function.

3. The intelligent agent trajectory navigation system based on visual language according to claim 1, characterized in that, The attention mechanism is a Transformer structure, which includes a multi-head attention layer, a feedforward sub-layer, residual connections and layer normalization, and applies position / time step encoding to temporal features to capture long-term dependencies.

4. A method for navigating the trajectory of an intelligent agent based on visual language, characterized in that, The navigation method is implemented based on the system described in any one of claims 1-3, and includes the following steps: S1. Obtain the historical trajectory sequence of at least one agent in the target scene. ,in Represents two-dimensional coordinates; S2. Based on the historical trajectory at each time step Constructing a time-series interaction graph intelligent agents Represented as a trajectory time-series interaction graph node Its state characteristics include position With speed And based on the distance between intelligent rows, it forms the edge set of the temporal interaction graph. ; S3. For any pair of adjacent nodes in the graph Calculate edge weights The node representations are updated in the graph convolutional network using the message passing rules modulated by the edge weights to obtain the dynamic interaction relationships of the agents; wherein: the edge weights It is given by the following formula: ； in: Let be the Euclidean distance between the rows. These are learnable parameters used to control interaction strength, decay rate, and range of influence; prediction interval length. Observation interval length and generate at least for each input sample Candidate future trajectories are selected to enhance multimodal coverage; S4. Train a graph convolutional network based on the dynamic interaction relationship of the target to obtain the dynamic interaction feature sequence of the agent; S5. Obtain the agent's motion trend trajectory sequence by training the dynamic interaction feature sequence of the agent based on the attention mechanism. ; S6. The optimized trajectory prediction module calculates and outputs the optimal agent motion trajectory based on the diversity-aware loss function for the agent's motion trend trajectory sequence; it jointly trains the parameters of the graph convolutional network and the time series prediction module based on a stochastic gradient descent-type optimizer, iterating in batches until convergence, and optionally employing an early stopping strategy to improve generalization performance, wherein: the diversity-aware loss is: 。

Images