Interactive correlation model and ship trajectory prediction method based on graph neural network

Abstract

The present invention provides an interactive correlation relationship model based on a graph neural network and a ship trajectory prediction method, which relate to the technical field of trajectory prediction, including: firstly, encoding the historical trajectory information of different types of intelligent agents through a multi-type intelligent agent encoding module to obtain the spatiotemporal characteristics of multiple intelligent agents; then constructing a spatiotemporal interaction graph based on the spatiotemporal characteristics of the multiple intelligent agents and the interactive correlation relationship between the intelligent agents, and integrating the interactive correlation information between the multiple intelligent agents into the update of the state of each intelligent agent through an interactive correlation relationship model based on a graph neural network; and then capturing the multimodal distribution of the future trajectory of the intelligent agent through a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder, and obtaining multiple possible future trajectories of each intelligent agent.

Description

Technical Field

[0001] The present invention relates to the technical field of trajectory prediction, and in particular to an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network. Background Art

[0002] Multi-agent trajectory prediction refers to the joint prediction of the future trajectories of all observed agents based on the historical trajectories of multiple agents in the scene and the map of the environment in which the agents are located. For example, in the scenario of autonomous driving, agents usually include traffic participants such as vehicles and pedestrians.

[0003] Traditional trajectory prediction tasks generally predict the future trajectory of a single agent in a scene. They can only integrate the appearance and motion characteristics of the agent, historical trajectory and simple map information, and cannot capture the interactive relationship between agents, making it difficult to accurately predict the future trajectories of multiple agents. However, in complex scenes, there is not only a single agent, but there are interactive movements between multiple agents. Therefore, the motion behavior of each agent not only determines its future trajectory, but the uncertainty of its future potential behavior will also have a certain impact on the motion behavior of other agents. At the same time, the motion behavior patterns of different types of agents are naturally different, and it is necessary to extract and fuse the data of different types of agents. In addition, since the future trajectory of the agent is uncertain and multimodal, trajectory prediction before this usually uses generative models to learn the distribution of future trajectories, such as conditional variational autoencoders (CVAE) and conditional generative networks (CGAN). These methods do not consider the correlation between samples, but randomly extract a set of future trajectories from the generative model, which will result in only a few modes being covered, which directly affects the safe operation of the multi-agent system. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network, so as to fully utilize the historical trajectory information and interactive correlation information of the intelligent agent and the multimodal output of the trajectory, so as to further improve the prediction accuracy.

[0005] A first aspect of an embodiment of the present invention provides an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network, the method comprising:

[0006] The historical trajectory information of different types of agents is encoded through the multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents;

[0007] According to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, a spatiotemporal interaction graph is constructed, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories;

[0008] Through the interactive correlation model based on graph neural network, we build interactive attention mechanisms at the agent level, category level and global level, integrate the interactive correlation into its own state update, and enhance the modeling ability of interactive relationships in multi-agent scenarios;

[0009] Through a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder, the multimodal distribution of the agent's future trajectory is captured, and multiple possible future trajectories of each agent are obtained.

[0010] Optionally, the multi-type agent encoding module is used to encode the historical trajectory information of different types of agents respectively to obtain the spatiotemporal characteristics of multiple agents, including:

[0011] In view of the fact that the motion behavior patterns of different types of intelligent agents are different, the trajectory data of intelligent agents with different motion patterns are encoded in a differentiated manner, including:

[0012] Standardize the spatiotemporal features of different dimensions of different categories of agents;

[0013] Different LSTM encoders are used to extract the spatiotemporal features of different types of agents through different network structure parameters. The structure parameters of the same type of agents are the same, while the structure parameters of different types of agents are different.

[0014] Optionally, constructing a spatiotemporal interaction graph according to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents includes:

[0015] According to the multiple intelligent agents, create multiple spatiotemporal nodes corresponding to the number of the multiple intelligent agents, each spatiotemporal node corresponding to one intelligent agent;

[0016] According to the spatial positions of the multiple agents, determining whether the distance between each two agents in the three-dimensional space is less than a set threshold;

[0017] When the distance between two agents in the three-dimensional space is less than the set threshold, a spatiotemporal edge is constructed between the nodes corresponding to the two agents;

[0018] According to the categories of the multiple agents, all agents of the same category are set as a category node, each category node corresponds to a category;

[0019] Build a category edge between two category nodes.

[0020] Optionally, the interactive association relationship model based on the graph neural network is used to construct the interactive attention mechanism at the agent level, category level and global level, and the interactive association relationship is integrated into the state update of the system, including:

[0021] Calculating agent-level interaction attention scores between spatiotemporal nodes of the spatiotemporal interaction graph;

[0022] Establishing an adjacency matrix according to the space-time edges of the space-time interaction graph, performing Laplace transform, and obtaining a Laplace normalized matrix;

[0023] Multiplying the agent-level interaction attention score by the Laplace normalized matrix to obtain an agent-level interaction matrix;

[0024] Calculating the category-level interaction attention scores between the category nodes of the spatiotemporal interaction graph, and obtaining the category-level interaction attention after normalization, which is also used as the category edge representing the interaction association relationship between the category nodes;

[0025] By performing an asymmetric convolution operation on the agent-level interaction matrix and then performing nonlinear activation, a global-level attention is obtained;

[0026] The category-level interaction attention is merged with the global-level attention to obtain the final attention of the spatiotemporal interaction graph, and the final attention is used as the adjacency matrix of the spatiotemporal interaction graph;

[0027] The interactive association relationship model based on the graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to achieve the state update of the node itself.

[0028] Optionally, the multimodal trajectory prediction module based on the learnable mapping sampling method and the conditional variational autoencoder is used to capture the multimodal distribution of the agent's future trajectory and obtain multiple possible future trajectories of each agent, including:

[0029] Mapping the features output by the interactive correlation model into latent variables through the learnable mapping-based sampling method;

[0030] The latent variables are decoded into K future trajectories of the agent through the conditional variational autoencoder.

[0031] Optionally, in the process of constructing the interactive attention mechanism at the agent level, category level and global level in the interactive association relationship model based on the graph neural network, the attention construction formula is:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, , Representative Category , The characteristics of || are concatenation operations. is the learnable attention weight vector, is a nonlinear activation function, is the maximum pooling operation, Representative Category With category The attention score between represents the category-level attention after weight distribution, is the global attention obtained from the agent-level attention, Represents the ultimate attention, The output of the interactive relationship model is Representation of layer interaction relationships.

[0037] A second aspect of an embodiment of the present invention provides an interactive correlation relationship model and a ship trajectory prediction device based on a graph neural network, the device comprising:

[0038] An encoding module, used to encode the historical trajectory information of different types of agents respectively through a multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents;

[0039] A spatiotemporal interaction graph construction module is used to construct a spatiotemporal interaction graph according to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories;

[0040] The spatiotemporal interaction graph calculation update module is used to build agent-level, category-level and global-level interaction attention mechanisms through an interaction correlation model based on a graph neural network, integrate the interaction correlation into its own state update, and enhance the modeling ability of interaction relationships in multi-agent scenarios;

[0041] The multimodal trajectory prediction module is used to capture the multimodal distribution of the agent's future trajectory through a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder, and obtain multiple possible future trajectories of each agent.

[0042] According to a third aspect of an embodiment of the present invention, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the interactive correlation relationship model and ship trajectory prediction method based on graph neural network as described in the first aspect.

[0043] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the interactive correlation relationship model and ship trajectory prediction method based on the graph neural network as described in the first aspect are implemented.

[0044] Beneficial effects of the present invention:

[0045] The embodiment of the present invention provides an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network, the method comprising: encoding the historical trajectory information of different types of agents respectively through a multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents, covering information such as the historical trajectory and category attributes of the agents; According to the spatiotemporal characteristics of the multiple agents, whether there is an interactive relationship between the multiple agents, and the categories of the multiple agents, a spatiotemporal interaction graph is constructed, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive relationship between agents of multiple categories; through the interactive relationship model based on the graph neural network, the interactive attention mechanism at the agent level, category level and global level is constructed, the interactive relationship is integrated into its own state update, and the modeling ability of the interactive relationship in the multi-agent scenario is enhanced; through the multimodal trajectory prediction module based on the learnable mapping sampling method and the conditional variational autoencoder, the multimodal distribution of the agent's future trajectory is captured, and multiple possible future trajectories of each agent are obtained. It can fully utilize and model the historical trajectory information and interactive information of the agent, and realize the multimodal output of the trajectory, further improving the accuracy of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0048] Figure 1 It is a flowchart of an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network provided by an embodiment of the present invention;

[0049] Figure 2 It is a framework diagram of an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network provided by an embodiment of the present invention;

[0050] Figure 3 It is a schematic diagram of a time-space interaction graph constructed in an interactive correlation relationship model based on a graph neural network and a ship trajectory prediction method provided by an embodiment of the present invention;

[0051] Figure 4 It is a framework diagram of another interactive correlation relationship model and ship trajectory prediction method based on graph neural network provided by one embodiment of the present invention;

[0052] Figure 5 It is a framework schematic diagram of an interactive correlation relationship model based on a graph neural network and a ship trajectory prediction device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Multi-agent trajectory prediction refers to the joint prediction of the future trajectories of all observed agents through the historical trajectories of multiple agents in the scene, as well as the map of the target agent's environment and other information. Agents usually include traffic participants such as vehicles and pedestrians. Traditional trajectory prediction tasks generally predict the future trajectory of a single agent in the scene, which only integrates the agent's appearance and motion characteristics, historical trajectory and simple map information.

[0056] However, in complex scenarios, there is not only a single agent, but there are interactive movements between multiple agents. Therefore, the movement behavior of each agent not only determines its future trajectory, but the uncertainty of its potential future behavior will also have a certain impact on the movement behavior of other agents. At the same time, the movement behavior patterns of different types of agents are naturally different, and the data of different types of agents need to be extracted and fused. In addition, since the future trajectory of the agent is uncertain and multimodal, trajectory prediction before this usually uses generative models to learn the distribution of future trajectories, such as conditional variational autoencoders (CVAE) and conditional generative networks (CGAN). These methods do not consider the correlation between samples, but randomly extract a set of future trajectories from the generative model, which results in only a few modes being covered, which directly affects the safe operation of the multi-agent system.

[0057] In order to effectively capture and model the motion information of multiple agents and the changes in their relationships, the technical solution of the present invention, based on the characteristics of multi-agent interaction scenarios, first designs a unified semantic feature extraction module to encode the historical trajectory of the agent and the position and motion information of the agent detected in the current frame. Then, a feature interaction relationship graph model based on a graph neural network is constructed to efficiently represent and learn the dynamic interaction relationship between agents. Finally, the multimodal trajectory prediction generation module learns a diversified sampling network to map the features of each agent to a set of potential codes, while covering the main modes and other modes in the CVAE latent space, thereby capturing the multimodal distribution of future trajectories.

[0058] Interactions between agents are ubiquitous, such as various interactions in the physical world, collision avoidance in autonomous driving, and confrontation and cooperation between players on the court. In multi-agent prediction tasks, exploring interactions between agents is the basis for accurate and explainable trajectory prediction, and these complex interactions have not been fully explored by previous work. Agents are affected by other agents in the same scene and change in real time, making it more difficult to capture such dynamic interactions.

[0059] Traditional trajectory prediction methods mainly include three categories:

[0060] ① Based on kinematic model: generally used to model the motion state of intelligent bodies and directly predict trajectories. Common models include Kalman filter model and Gaussian mixture model. It has high accuracy in short-term trajectory prediction, but is relatively sensitive to noise and has poor robustness in long-term trajectory prediction.

[0061] ② Based on probability model: mainly predict the behavior intention of intelligent agents through Bayesian networks, decision trees, etc.

[0062] ③Combination of the first two types of comprehensive methods.

[0063] Since traditional trajectory prediction methods simulate the interactive behaviors of multiple agents through artificially designed features, they only obtain simple interactive features between multiple agents, but do not utilize the structural characteristics of the trajectory and the interactive relationship characteristics between multiple agents.

[0064] In one embodiment:

[0065] Figure 1 is a flowchart of an interactive correlation relationship model and a ship trajectory prediction method based on a graph neural network provided by an embodiment of the present invention. Figure 2 1 is a framework diagram of an interactive correlation model and a ship trajectory prediction method based on a graph neural network provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides an interactive correlation relationship model based on a graph neural network and a ship trajectory prediction method for predicting the future trajectory of a ship, the method comprising:

[0066] In step S101, the historical trajectory information of different types of agents is encoded respectively by a multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents;

[0067] In step S102, a spatiotemporal interaction graph is constructed based on the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories;

[0068] In step S103, through the interactive association model based on the graph neural network, the interactive attention mechanism at the agent level, category level and global level is constructed, the interactive association relationship is integrated into its own state update, and the modeling ability of the interactive relationship in the multi-agent scenario is enhanced;

[0069] In step S104, a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder is used to capture the multimodal distribution of the agent's future trajectory and obtain multiple possible future trajectories of each agent.

[0070] First, in step S101, the multi-type agent encoding module is a module that processes the historical trajectory information of multi-type agents. Different types of agents include pedestrians, motor vehicles, non-motor vehicles, etc. By encoding the historical trajectory information of different types of agents through the multi-type agent encoding module, the spatiotemporal representation of these different types of agents can be obtained. The spatiotemporal representation can represent the motion state pattern characteristics of different types of agents.

[0071] Further, in step S102, a spatiotemporal interaction graph is constructed based on the spatiotemporal characteristics of multiple different types of agents obtained in step S101. The spatiotemporal interaction graph is used to represent the interactive association relationships between different types of agents and the interactive association relationships between categories of different types of agents, that is, the interactive association relationships between agents and the interactive association relationships between categories.

[0072] Specifically, first, based on the obtained spatiotemporal characteristics of multiple different types of agents, it is determined whether there is an interactive correlation relationship between multiple different types of agents, and the category correlation relationship between the categories of multiple different types of agents. Further, based on the interactive correlation relationship between the above-mentioned agents and the interactive correlation relationship between the categories of agents, a spatiotemporal interaction graph is constructed. In this spatiotemporal interaction graph, multiple spatiotemporal nodes correspond to multiple observed agents, and spatiotemporal edges are used to represent the interactive correlation relationship between agents. At the same time, the spatiotemporal interaction graph also includes category nodes, which are used to represent multiple agents belonging to the same category, and category edges are used to represent the interactive correlation relationship between agents of multiple categories. The construction of the spatiotemporal interaction graph can effectively display the interactive correlation relationship between different agents and the interactive correlation relationship between the categories to which different agents belong.

[0073] Further, in step S103, the embodiment of the present application uses the interactive association relationship model based on the graph neural network to construct the interactive attention mechanism at the agent level, category level and global level, and integrates the interactive relationship between the agents into the state update of the agent itself, so as to enhance the modeling ability of the interactive relationship in the multi-agent scenario. Specifically, the spatiotemporal interaction graph constructed above is input into the interactive association relationship model based on the graph neural network, and the interactive association relationship between multiple different types of agents is extracted. Further, the interactive attention mechanism is introduced into the interactive association relationship model based on the graph neural network, and the interactive relationship is paid attention to and learned at the agent level, category level and global level respectively. In this way, the interactive association relationship between multiple different types of agents can be effectively captured and modeled at different levels. Finally, the interactive attention mechanism is used to integrate the interactive association relationship between multiple different types of agents into the state update of multiple different types of agents themselves. In this way, multiple different types of agents can better consider the interactive association relationship with other agents in the process of state update, thereby improving the interactive association relationship modeling ability of the interactive association relationship model based on the graph neural network in the multi-agent scenario.

[0074] Finally, in step S104, by combining the learnable mapping sampling method and the conditional variational autoencoder, a multimodal trajectory prediction module is constructed to capture the multimodal distribution of the future trajectories of multiple different types of agents, so as to obtain multiple possible future trajectories of each agent in multiple different types of agents. Specifically, a trajectory prediction module is constructed by combining the learnable mapping sampling method and the conditional variational autoencoder. The learnable mapping sampling method is used to sample and generate trajectory samples from a latent space, while the conditional variational autoencoder is used to represent the trajectory data as a probability distribution of the latent space. Further, the multimodal distribution of the future trajectory of the agent is captured. That is, there may be multiple possibilities for the future trajectory of the agent, and each possibility has a certain probability of occurrence. Therefore, through the multimodal trajectory prediction module, multiple future trajectories that may exist in the future for each agent can be obtained, and each future trajectory corresponding to each agent reflects a predicted trajectory direction. In this way, when facing the problem of predicting the future trajectory of the agent, multiple possible future trajectories can be considered and compared to improve the accuracy and comprehensiveness of the prediction of the trajectory of the agent.

[0075] Optionally, the step S101 includes:

[0076] In view of the fact that the motion behavior patterns of different types of intelligent agents are different, the trajectory data of intelligent agents with different motion patterns are encoded in a differentiated manner, including:

[0077] Standardize the spatiotemporal features of different dimensions of different categories of agents;

[0078] Different LSTM (Long Short-Term Memory) encoders are used to extract the spatiotemporal characteristics of different types of agents through different network structure parameters. The structure parameters of the same type of agents are the same, while those of different types of agents are different.

[0079] Specifically, in one embodiment, in view of the fact that the motion behavior patterns of different types of intelligent agents are different, the trajectory data of intelligent agents with different motion patterns can be encoded in a differentiated manner. For example, pedestrians, motor vehicles, and non-motor vehicles are intelligent agents with different motion behavior patterns. Therefore, different methods can be used to encode intelligent agents with different motion behavior patterns. For example, different encoders can be used to encode the trajectory data (the trajectory data includes historical trajectory information) of intelligent agents with different motion behavior models, such as Figure 2 As shown, in the above embodiment, the multi-type agent encoding module includes multiple encoders, through the multiple encoders (i.e. Figure 2 The encoder 1~encoder M in the above example can be used to analyze the historical trajectory information of multiple agents of different types (i.e. Figure 2 The multi-agent historical trajectories in the multi-type agent encoding module are encoded to obtain the spatiotemporal characteristics of multiple agents. Specifically, for the historical trajectory information of different types of agents, multiple encoders with different parameters in the multi-type agent encoding module are used for encoding processing to obtain the spatiotemporal characteristics of each agent. The multi-type agent encoding module can perform different encoding processing on its historical trajectory information according to the type or feature difference of the agent. Furthermore, the spatiotemporal features of different dimensions of different categories of agents are standardized to ensure that the various types of agents have consistency in the spatiotemporal feature representation; different LSTM encoders are used to differentially extract the spatiotemporal features of different types of agents through different network structure parameters. Specifically, the same type of agent uses the same encoder structure parameters, while different types of agents use different encoder structure parameters to ensure personalized processing of feature extraction of different types of agents, thereby improving the accuracy of spatiotemporal features of different types of agents.

[0080] By encoding the historical trajectory information of different types of agents through multi-type agent encoding modules and differentiated processing methods to obtain the spatiotemporal characteristics of different types of agents, the characteristic differences between different types of agents can be effectively characterized, thereby improving the behavior prediction and analysis capabilities of multi-type agents.

[0081] Optionally, the step S102 includes:

[0082] According to the multiple intelligent agents, create multiple spatiotemporal nodes corresponding to the number of the multiple intelligent agents, each spatiotemporal node corresponding to one intelligent agent;

[0083] According to the spatial positions of the multiple agents, determining whether the distance between each two agents in the three-dimensional space is less than a set threshold;

[0084] When the distance between two agents in the three-dimensional space is less than the set threshold, a spatiotemporal edge is constructed between the nodes corresponding to the two agents;

[0085] According to the categories of the multiple agents, all agents of the same category are set as a category node, each category node corresponds to a category;

[0086] Build a category edge between two category nodes.

[0087] Specifically, in the embodiment of the present application, a spatiotemporal interaction graph (such as a spatiotemporal interaction graph) can be constructed according to the spatiotemporal characteristics of multiple different types of intelligent agents, the interactive association relationship between multiple different types of intelligent agents, and the different categories of multiple different types of intelligent agents. Figure 3 shown).

[0088] First, according to multiple agents, multiple spatiotemporal nodes corresponding to the number of multiple agents are created, and each spatiotemporal node corresponds to an agent. Further, the positions of the multiple agents are unified in the same coordinate system to obtain the spatial position of each agent. According to the spatial positions of the multiple agents, it is determined whether the distance between each two agents in the three-dimensional space is less than the set threshold. The set threshold can be set according to the actual situation. Further, when the distance between the two agents in the three-dimensional space is less than the set threshold, a spatiotemporal edge is constructed between the corresponding spatiotemporal nodes of the two agents in the spatiotemporal interaction graph, indicating that there is an interactive relationship between the two agents whose distance is less than the set threshold. Finally, according to the categories of the multiple agents, the multiple agents belonging to the same category are set as a category node in the spatiotemporal interaction graph, and each category node corresponds to the category of an agent. A category edge is constructed between two category nodes to indicate the association relationship between the categories of different agents.

[0089] By constructing such a spatiotemporal interaction graph, we can better understand and analyze the interaction relationship between intelligent agents, and apply it to areas such as behavior prediction and path planning in practical applications.

[0090] Figure 3 It is a schematic diagram of a space-time interaction graph constructed in an interactive correlation relationship model based on a graph neural network and a ship trajectory prediction method provided by an embodiment of the present invention, including space-time nodes, space-time edges, category nodes and category edges.

[0091] like Figure 3 As shown, Figure 3 The multiple solid black circles in Figure 3 ) represents multiple agents of different categories. Some of the multiple agents have the same category, while others have different categories. Multiple agents of the same category are Figure 3 In the figure, the agents are bounded by the dotted ellipse boundary, and the interaction relationship between the agents is the space-time edge. It can be understood that there may be an interaction relationship between two agents or there may not be an interaction relationship between them. Figure 3 This is a schematic diagram, taking the interactive relationship between two agents as an example. Figure 3 In the case of an agent being associated with another agent, a spatiotemporal edge is generated between the agent and the other agent (in Figure 3 In the figure, the agents of different categories will also have interactive relationships between categories, that is, interactive relationships between agents of multiple categories. Therefore, Figure 3 In , each category generates a category edge with other categories (in Figure 3 It is understandable that there may or may not be an interactive relationship between two agent categories. Figure 3 This is a schematic diagram, taking the interactive relationship between two agent categories as an example.

[0092] Optionally, the step S103 includes:

[0093] Calculating agent-level interaction attention scores between spatiotemporal nodes of the spatiotemporal interaction graph;

[0094] Establishing an adjacency matrix according to the space-time edges of the space-time interaction graph, performing Laplace transform, and obtaining a Laplace normalized matrix;

[0095] Multiplying the agent-level interaction attention score by the Laplace normalized matrix to obtain an agent-level interaction matrix;

[0096] Calculating the category-level interaction attention scores between the category nodes of the spatiotemporal interaction graph, and obtaining the category-level interaction attention after normalization, which is also used as the category edge representing the interaction association relationship between the category nodes;

[0097] By performing an asymmetric convolution operation on the agent-level interaction matrix and then performing nonlinear activation, a global-level attention is obtained;

[0098] The category-level interaction attention is merged with the global-level attention to obtain the final attention of the spatiotemporal interaction graph, and the final attention is used as the adjacency matrix of the spatiotemporal interaction graph;

[0099] The interactive association relationship model based on the graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to achieve the state update of the node itself.

[0100] Specifically, Figure 2 As shown, in an embodiment of the present application, an interactive attention mechanism of agent-level attention, category-level attention and global-level attention can be constructed through an interactive association relationship model based on a graph neural network, and the interactive association relationship is integrated into its own state update. First, the agent-level interactive attention score is calculated according to the spatiotemporal nodes of the spatiotemporal interaction graph. The agent-level interactive attention score represents the interactive attention between different agents and is used to measure the degree of interactive association between different agents. Further, an adjacency matrix is ​​established according to the spatiotemporal edges of the spatiotemporal interaction graph, a Laplace transform is performed, and a Laplace normalized matrix is ​​obtained. Then the agent-level interactive attention score is multiplied by the Laplace normalized matrix to obtain an agent-level interactive matrix, which is used to represent the interactive relationship between agents. Further, the category-level interactive attention score between the category nodes of the spatiotemporal interaction graph is calculated. The category-level interactive attention score represents the interactive attention between different categories. After normalization, the category-level interactive attention is obtained, and it is also used as a category edge representing the interactive association relationship between category nodes. Furthermore, by performing an asymmetric convolution operation on the agent-level interaction matrix and then performing nonlinear activation, we obtain global-level attention for modeling global interaction relationships. Furthermore, we fuse the category-level interaction attention with the global-level attention to obtain the final attention of the spatiotemporal interaction graph, and use it as the adjacency matrix of the spatiotemporal interaction graph for state updates of the spatiotemporal nodes corresponding to the agent.

[0101] The interactive association relationship model based on graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to realize the state update of spatiotemporal nodes themselves.

[0102] Optionally, the step S104 includes:

[0103] Mapping the features output by the interactive correlation model into latent variables through the learnable mapping-based sampling method;

[0104] The latent variables are decoded into K future trajectories of the agent through the conditional variational autoencoder.

[0105] Specifically, Figure 2As shown, in an embodiment of the present application, the multimodal distribution of the agent's future trajectory, that is, different possible future trajectories, can be captured based on a multimodal trajectory prediction module of a learnable mapping sampling method and a conditional variational autoencoder. First, the spatiotemporal features output by the interactive correlation model based on the graph neural network are mapped to latent variables through a learnable mapping sampling method. That is, the spatiotemporal feature representation extracted by the interactive correlation model based on the graph neural network is converted into a potential latent variable space. Then, through the conditional variational autoencoder ( Figure 2 The latent variables obtained by mapping the spatiotemporal features of each agent's state update are decoded into K future trajectories of the agent, and each decoder outputs a future trajectory. In this process, the conditional variational autoencoder is used to map the learned latent variables into a multimodal distribution of the agent's future trajectory, thereby obtaining multiple possible future trajectories. Figure 2 The multi-agent trajectory prediction shown ( Figure 2 lower right corner), in Figure 2 In the figure, the dotted line represents the predicted future trajectory of the multi-agent, and the solid line represents the historical trajectory of the multi-agent.

[0106] Therefore, the learnable mapping sampling method and conditional variational autoencoder are used to realize the multimodal prediction of the agent's future trajectory. Through the learned mapping and decoding, multiple possible future trajectory distributions of the agent can be obtained, which improves the accuracy of predicting the future direction of the agent's behavior.

[0107] Optionally, in the process of constructing the interactive attention mechanism at the agent level, category level and global level in the interactive association relationship model based on the graph neural network, the attention construction formula is:

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, , Represents the category characteristics, is a nonlinear activation function, is the learnable attention weight vector, is the maximum pooling operation, Representative Category With category The category-level interaction attention score between , || is the concatenation operation, represents the category-level attention after weight distribution, is the global attention obtained from the agent-level attention, Represents the ultimate attention, The output of the interactive relationship model is Layer interaction relationships.

[0113] In another embodiment:

[0114] In step S201, the historical trajectory information of M agents detected from the historical frame is encoded, and the current trajectory information of N agents detected from the current frame is encoded to obtain the semantic features of M+N agents;

[0115] In step S202, based on the semantic features of the M+N agents, whether there is an interactive relationship between the M agents in the historical frame, whether there is an interactive relationship between the N agents in the current frame, and whether the M agents in the historical frame can match the N agents in the current frame, a graph structure with M+N nodes is constructed, each node in the graph structure corresponds to a semantic feature of an agent, and each edge in the graph structure represents an interactive relationship between two agents;

[0116] In step S203, an interactive association relationship model based on an L-layer graph neural network is constructed, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in the current frame. The semantic features of the M+N agents are updated through the interactive association relationship model to obtain updated semantic features of the agents. Updated edge features are obtained based on the updated semantic features. The edge connections are dynamically changing in time, so that the GNN can simulate interactions between different numbers of objects in different scenes.

[0117] In step S204, the historical trajectory information of the M agents detected in the historical frame, the updated semantic features of the M agents, and the updated edge features are input into a multimodal trajectory prediction module based on a conditional variational autoencoder to obtain multiple future trajectory information of the M agents.

[0118] First, in step S201, the historical trajectory information of M agents detected from the historical frame is encoded, and the current trajectory information of N agents detected from the current frame is encoded to obtain the semantic features of M+N agents.

[0119] In the embodiment of the present invention, the intelligent agent is an object of different types with motion information, such as pedestrians, motor vehicles, bicycles, etc. Before collecting the trajectory information of the intelligent agent, it is assumed that the current time is Time, the historical frame is the current For each frame within a preset number of time steps before the moment, the historical trajectory information of all agents contained in the historical frame can be expressed as ,current The moment is the current frame, and the current trajectory information of the N agents detected in the current frame The current trajectory information includes the position, direction, speed and other information of each of the N agents, and the historical trajectory information includes the position, direction, speed and other information of each of the M agents. Due to the dynamic changes of the agents, the number of agents included in the historical frame is not necessarily the same as the number of agents included in the current frame. Therefore, the number of agents included in the historical frame is set to M, and the number of agents included in the current frame is set to N.

[0120] After obtaining the historical trajectory information of M agents detected in the historical frame and the current trajectory information of N agents detected in the current frame, the historical trajectory information of M agents detected from the historical frame and the current trajectory information of N agents detected in the current frame are encoded by the encoder respectively to obtain the semantic features of M+N agents.

[0121] Further, in step S202, based on the semantic features of the M+N agents, whether there is an interactive relationship between the M agents in the historical frame, whether there is an interactive relationship between the N agents in the current frame, and whether there is an interactive relationship between the M agents in the historical frame and the N agents in the current frame, a graph structure with M+N nodes is constructed, each node in the graph structure corresponds to a semantic feature of an agent, and each edge in the graph structure represents the existence of an interactive relationship between two agents.

[0122] In an embodiment of the present invention, after obtaining the semantic features of M+N agents, a graph structure is constructed based on the semantic features of the M+N agents, whether there is an interactive relationship between the M agents in the historical frame, whether there is an interactive relationship between the N agents in the current frame, and whether there is an interactive relationship between the M agents in the historical frame and the N agents in the current frame. The interactive relationship is the interactive association between any two agents in the same frame. The interactive association between agents is reflected between adjacent agents. The graph structure constructed by the above content contains M+N nodes. Each node in the graph structure corresponds to a semantic feature obtained by encoding an agent. The interactive relationship between two agents is represented in the graph structure by the edges between the nodes.

[0123] Further, in step S203, an interactive association relationship model based on an L-layer graph neural network is constructed, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in current frames. The semantic features of the M+N agents are updated through the interactive association relationship model to obtain updated semantic features of the agents, and updated edge features are obtained based on the updated semantic features. The edge connections change dynamically in time so that the GNN can simulate interactions between different numbers of objects in different scenes.

[0124] In an embodiment of the present invention, after constructing a graph structure, an interactive association relationship model based on an L-layer graph neural network is constructed, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in current frames, and the semantic features of the M+N agents in the graph structure having M+N nodes in the interactive association relationship model are updated with the semantic features of the M agents in the historical frames and the semantic features of the N agents in the current frame. After the update, the updated semantic features of the M agents in the historical frames and the updated semantic features of the N agents in the current frame are obtained, and based on the updated semantic features of the M agents in the historical frames and the updated semantic features of the N agents in the current frame, the updated edge features are obtained, that is, each agent has the updated semantic features after the update, and the edge features of the edges between the nodes corresponding to the two agents are updated by the updated semantic features of the M agents in the historical frames and the updated semantic features of the N agents in the current frame to obtain the updated edge features.

[0125] Finally, in step S204, the updated semantic features of the M agents are input into a multimodal trajectory prediction module based on a conditional variational autoencoder to obtain multiple future trajectory information of the M agents.

[0126] In an embodiment of the present invention, after obtaining the updated semantic features of M agents in the historical frame and the updated semantic features of N agents in the current frame and the updated edge features, the updated semantic features of the M agents are further input into a multimodal trajectory prediction module based on a conditional variational autoencoder, and multiple future trajectory information of the M agents is output through the multimodal trajectory prediction module based on the conditional variational autoencoder, that is, each of the M agents corresponds to multiple future trajectory information in the future time, and the future trajectory information is the trajectory route of the agent.

[0127] Through the above embodiments, it is possible to encode the historical trajectory information and the current trajectory information of the intelligent agent to obtain the semantic features of multiple intelligent agents, construct a graph structure based on the semantic features of the multiple intelligent agents and the interaction relationship between the multiple intelligent agents, update the semantic features of the multiple intelligent agents through an interactive association relationship model based on a graph neural network to obtain updated semantic features, and then input the updated semantic features into a multimodal trajectory prediction module based on a conditional variational autoencoder to obtain the future trajectory information of the multiple intelligent agents, thereby updating the semantic features through the interactive association relationship model based on the interaction relationship between the multiple intelligent agents, and capturing the multimodal distribution of the future trajectory of the intelligent agent through the multimodal trajectory prediction model, and obtaining multiple possible future trajectories of each intelligent agent in the future time.

[0128] Optionally, the step S201 includes:

[0129] Inputting historical trajectory information of the M agents in historical frames before the current moment into a long short-term memory artificial neural network, reading the time series of the historical frames through the long short-term memory artificial neural network, and performing time dynamic modeling on the time series to obtain historical trajectory features of the M agents, wherein agents of different categories among the M agents are respectively input into encoders of long short-term memory artificial neural networks with different parameter structures for encoding, and agents of the same type are input into encoders of long short-term memory artificial neural networks with the same parameter structure for encoding;

[0130] Specifically, in the above embodiment, when encoding the historical trajectory information of M agents detected from the historical frames, considering that the motion patterns of different types of agents are naturally different, different encoders are designed for the trajectory information of different types of agents to extract semantic features.

[0131] For the historical trajectory information of the M agents contained in the historical frame before the current moment, the encoder of the long short-term memory artificial neural network (LSTM) is selected for encoding. In the encoding process of the historical trajectory information of the M agents contained in the historical frame, the M agents are input into the encoder of the long short-term memory artificial neural network (LSTM) with different parameter structures according to categories. The time series of the M agents in the historical frame are read through the encoder of the long short-term memory artificial neural network (LSTM) with different parameter structures, and the time series of the historical frame is modeled dynamically to obtain the historical trajectory characteristics of the M agents. ,in, The historical trajectory features obtained by encoding the historical trajectory information of M agents respectively enable different categories of agents among the M agents to be encoded in encoders of long short-term memory artificial neural networks with different parameter structures, while the same type of agents are input into the encoder of the long short-term memory artificial neural network with the same parameter structure for encoding.

[0132] The current trajectory information of the N agents in the current frame is encoded by a multilayer perceptron. Specifically, in the encoding process of the current trajectory information of the N agents in the current frame, the current trajectory information of the N agents in the current frame is input into the multilayer perceptron (MLP), and the current trajectory features of the N agents are output through the multilayer perceptron (MLP) mapping. ,in, The current trajectory features are obtained by encoding the current trajectory information of N agents respectively.

[0133] After obtaining the historical trajectory characteristics of M agents and the current trajectory characteristics of N agents Afterwards, these two are used as semantic features of M+N agents.

[0134] like Figure 4 As shown, the historical trajectory information of the M agents contained in the historical frame is input into multiple long short-term memory artificial neural networks with different parameter structures ( Figure 4 In the long short-term memory artificial neural network 1~long short-term memory artificial neural network M), the current trajectory information of the N agents contained in the current frame is input into multiple multilayer perceptrons with different parameter structures ( Figure 4 In the figure, multiple long short-term memory artificial neural networks with different parameter structures are used to output the semantic features of M agents. Multiple multilayer perceptrons with different parameter structures are used to output the semantic features of N agents, that is, the semantic features of M+N agents are obtained. The semantic features of M+N agents are used to construct a graph structure with M+N nodes. The graph structure with M+N nodes is processed by an interactive correlation relationship model with L layers to realize the iterative update of the semantic features of M+N agents, and the updated semantic features and updated edge features of M+N agents are obtained. For the trajectory prediction of the ith agent, the updated semantic features of the ith agent are input into the multimodal trajectory prediction module based on the conditional variational autoencoder to obtain multiple ( Figure 4 (Take K as an example) future trajectory information.

[0135] Optionally, the step S202 includes:

[0136] According to the trajectory information of the M agents detected in the historical frame and the position information of the N agents detected in the current frame, M+N nodes are created, each node corresponding to the semantic feature of an agent;

[0137] According to the semantic features of the M+N agents, determining whether the distance between every two agents in the three-dimensional space is less than a set threshold;

[0138] When the distance between the two agents in the three-dimensional space is less than a set threshold, an edge is constructed between the nodes corresponding to the two agents.

[0139] Specifically, in an embodiment of the present invention, when constructing a graph structure with M+N nodes, it is first necessary to create M+N nodes based on the semantic features of M agents in the historical frame and the semantic features of N agents in the current frame among the semantic features of the M+N agents, so that each of the M+N nodes corresponds to the semantic features of one of the M+N agents. After obtaining the semantic features of the M+N agents, it is further determined based on the semantic features of the M+N agents whether the distance between every two of the M+N agents in the three-dimensional space is less than a set threshold. When there are two agents whose distance in the three-dimensional space is less than the set threshold, it indicates that there is an interactive relationship between the two agents. Therefore, an edge is constructed between the nodes corresponding to the two agents with an interactive relationship.

[0140] It should be noted that when judging the distance between intelligent agents in three-dimensional space, it is necessary not only to judge the relationship between the distance between the intelligent agents included in the M intelligent agents and the N intelligent agents and the set threshold, but also to judge the relationship between the distance between any two intelligent agents among the M intelligent agents and the set threshold, and the relationship between the distance between any two intelligent agents among the N intelligent agents and the set threshold, that is, it is necessary to consider the semantic features of M+N intelligent agents at the same time, whether there is an interactive relationship between the M intelligent agents in the historical frame, whether there is an interactive relationship between the N intelligent agents in the current frame, and whether there is an interactive relationship between the M intelligent agents in the historical frame and the N intelligent agents in the current frame.

[0141] Through the above embodiments, the interactive associations between intelligent agents can be reflected in the graph structure through the edges between the nodes corresponding to the intelligent agents, which is conducive to the efficient learning of the graph neural network.

[0142] Optionally, the step S203 includes:

[0143] The interactive association relationship model includes L layers, the semantic features of the M+N agents are updated L times by iteration, and the edge features between the M agents in the historical frame and the N agents in the current frame are updated L times by iteration;

[0144] The semantic features of the M+N agents at the L layer are used as updated semantic features of the M agents and updated semantic features of the N agents in the current frame, and the difference between the semantic features of the M agents at the L layer in the historical frame and the semantic features of the N agents at the L layer in the current frame is used as the updated edge feature;

[0145] Among them, the first The iteration update is:

[0146] For the agent in The semantic features of the layer and the corresponding agents connected by edges are The semantic features of the layer are aggregated to obtain the agent in the The semantic features of the layer are used to transform the M agents in the historical frame into The semantic features of the layer are related to the N agents in the current frame. The difference in semantic features between the layers is used as the Update the subsequent features in the next iteration.

[0147] Specifically, in an embodiment of the present invention, an interactive association relationship model based on an L-layer graph neural network is constructed, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in current frames, and the semantic features of M+N agents are updated through the interactive association relationship model, wherein the interactive association relationship model includes L layers, and the semantic features of M+N agents are iteratively updated L times through the L-layer interactive association relationship model, and the edge features between the M agents in the historical frame and the N agents in the current frame are iteratively updated L times, and after each update, the semantic features of the nodes in the latter layer have the semantic features of the nodes in each layer before the layer, and the edge features in the latter layer have the edge features of each layer before the layer.

[0148] Specifically, in the iterative updating process, the semantic features of M+N agents at the L layer are used as the updated semantic features of the M agents in the historical frame and the updated semantic features of the N agents in the current frame, and the difference between the updated semantic features of the M agents at the L layer in the historical frame and the updated semantic features of the N agents at the L layer in the current frame is used as the updated edge features of the Lth layer.

[0149] In the following, we take the The iterative update process is described using the layer as an example, where 0< <L:

[0150] In the L-th iteration update The next iteration update is to update the agent in the The semantic features of the layer include The semantic features of M agents in the historical frame corresponding to the nodes of the layer and The semantic features of N agents in the current frame corresponding to the nodes of the layer , and the agents corresponding to the nodes connected by the edges are in The semantic features of the layer are aggregated to obtain the agent in the The semantic features of the layer are used to transform the M agents in the historical frame into The semantic features of the layer are related to the N agents in the current frame. The difference in semantic features between the layers is used as the The next iteration updates the following features, i.e. in, For the The updated feature after the L-th iteration is: .

[0151] Optionally, the pair of agents The semantic features of the layer, the agents corresponding to the nodes connected by the edges are in the The semantic features of the layer are aggregated to obtain the agent in the Semantic features of the layer, including:

[0152] For the mth agent among the M agents, The semantic features of the layer, the agent that has an interactive relationship with the mth agent among the M agents in the historical frame, The semantic features of the layer, and the agent in the current frame that has an interactive relationship with the mth agent among the N agents in the current frame The semantic features of the layer are aggregated to obtain the agent in the Semantic features of layers;

[0153] For the nth agent among N agents, The semantic features of the layer, the agent that has an interactive relationship with the nth agent in the current frame The semantic features of the layer, and the agent that has an interactive relationship with the nth agent in the M agents in the historical frame The semantic features of the layer are aggregated to obtain the agent in the Semantic features of the layer.

[0154] Specifically, in the embodiment of the present invention, in In the iteration update, The semantic features of each agent contained in the layer are iteratively updated. For example, The mth agent among the M agents in the historical frame contained in the layer is the mth agent in the The semantic features of the layer, the agent that has an interactive relationship with the mth agent among the M agents in the historical frame (i.e. Among the M nodes in the historical frame in the layer, the agent corresponding to other nodes connected to the node corresponding to the mth agent through edges) The semantic features of the layer, as well as the agents that have an interactive relationship with the mth agent among the N agents in the current frame (i.e., in Among the N nodes in the current frame in the layer, the agent corresponding to other nodes connected to the node corresponding to the mth agent through edges) is in the The semantic features of the layer are aggregated to obtain the mth agent in the The semantic features of the layer, where the mth agent is any one of the M agents in the historical frame; for the nth agent among the N agents, the nth agent in the first The semantic features of the layer, the agent that has an interactive relationship with the nth agent among the N agents in the current frame (i.e., Among the N nodes in the current frame in the layer, the agent corresponding to other nodes connected to the node corresponding to the nth agent through edges) is in the The semantic features of the layer, and the agents among the M agents in the historical frame that have an interactive relationship with the nth agent (i.e., Among the M nodes in the historical frame in the layer, the agent corresponding to other nodes connected to the node corresponding to the nth agent through edges) The semantic features of the layer are aggregated to obtain the nth agent in the The semantic features of the layer, where the nth agent is any one of the N agents in the current frame.

[0155] Optionally, the step S204 includes:

[0156] Inputting the updated semantic features of the M agents into a multimodal trajectory prediction module based on a conditional variational autoencoder to generate hidden variables corresponding to each of the M agents;

[0157] According to the hidden variables corresponding to each of the M intelligent agents, multiple future trajectory information of the M intelligent agents is obtained.

[0158] Specifically, in the embodiment of the present invention, after obtaining the updated semantic features and the updated edge features of the M agents, the updated semantic features of the M agents are input into the multimodal trajectory prediction module based on the conditional variational autoencoder to generate the hidden variables corresponding to each of the M agents, and through the hidden variables corresponding to each of the M agents, multiple future trajectory information of the M agents is obtained. The hidden variables can simulate factors such as the behavior intention of the agent, capture the multimodal distribution of the future trajectory of the M agents, and obtain multiple possible trajectory routes of the M agents in the future.

[0159] Figure 4 It is a framework diagram of another interactive correlation model and ship trajectory prediction method based on a graph neural network provided by an embodiment of the present invention.

[0160] Specifically, Figure 4 As shown in Figure 1, a conditional variational autoencoder (CVAE) is used as the generative model, and a set of hidden variables are introduced to capture the multimodal distribution of future trajectories. The encoder outputs the updated semantic features of the M agents based on the interactive correlation model. and future trajectory , infer the posterior distribution of the latent variable , in the latent variable The corresponding mean and variance are estimated under the assumption that the distribution is normal. The decoder generates a sample distribution similar to the input by sampling from this distribution through multimodal mapping. Among them, a set of hidden variables for each of the M agents is , corresponding to the future trajectory of the ith agent ,in, They respectively represent the k future trajectory information of the i-th agent.

[0161] In summary, the interactive correlation model and ship trajectory prediction method based on graph neural network provided by the present invention can achieve the following objectives:

[0162] ① Aiming at the problem of simultaneous interactive motion of multiple agents in complex scenarios, a multi-agent trajectory prediction method framework is proposed based on an interactive correlation graph network model and a multimodal trajectory prediction module based on conditional variational autoencoder.

[0163] ② In view of the fact that the motion behavior patterns of different types of intelligent agents have natural differences, encoders with different parameter structures are set to encode the trajectory information of intelligent agents with different motion patterns in a differentiated manner.

[0164] ③ In view of the fact that the interaction correlation between intelligent agents is highly random and difficult to capture regularities, an interaction correlation relationship model based on graph neural networks is proposed, and a semantic feature interaction mechanism based on graph neural networks is introduced to form an efficient collaborative semantic feature aggregation process, thereby improving the relationship modeling capabilities of multiple agents in interactive scenarios.

[0165] ④ In order to address the problem that the random sampling results of the deep random generation model are uncontrollable without considering the correlation between samples, it is proposed to learn a diversified sampling network through a multimodal trajectory prediction generation module to map the features of each agent to a set of latent codes, while covering the main modes and other modes in the CVAE latent space, thereby capturing the multimodal distribution of future trajectories.

[0166] Figure 5 is a schematic diagram of a framework of an interactive correlation relationship model and a ship trajectory prediction device based on a graph neural network provided by an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0167] The encoding module 11 is used to encode the historical trajectory information of different types of agents respectively through the multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents;

[0168] A spatiotemporal interaction graph construction module 12 is used to construct a spatiotemporal interaction graph according to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories;

[0169] The spatiotemporal interactive attention calculation and update module 13 is used to construct the interactive attention mechanism at the agent level, category level and global level through the interactive correlation model based on the graph neural network, integrate the interactive correlation into its own state update, and enhance the modeling ability of the interactive relationship in the multi-agent scenario;

[0170] The multimodal trajectory prediction module 14 is used to capture the multimodal distribution of the future trajectory of the agent through a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder, and obtain multiple possible future trajectories of each agent.

[0171] The encoding module 11 includes:

[0172] The encoding unit is used to encode the trajectory data of the agents with different motion patterns in a differentiated manner in view of the fact that the motion behavior patterns of different types of agents are different, including:

[0173] The standardization unit is used to standardize the spatiotemporal features of different dimensions of different categories of agents;

[0174] The spatiotemporal feature extraction unit is used to use different LSTM encoders to differentially extract the spatiotemporal features of different types of agents through different network structure parameters, wherein the structure parameters of the same type of agents are the same, and the structure parameters of different types of agents are different.

[0175] The spatiotemporal interaction graph construction module 12 includes:

[0176] A creation module, used for creating a plurality of spatiotemporal nodes corresponding to the number of the plurality of intelligent agents according to the plurality of intelligent agents, each spatiotemporal node corresponding to one intelligent agent;

[0177] A judgment module, used for judging whether the distance between every two intelligent agents in the three-dimensional space is less than a set threshold according to the spatial positions of the multiple intelligent agents;

[0178] A first construction module is used to construct a spatiotemporal edge between nodes corresponding to the two agents when the distance between the two agents in the three-dimensional space is less than the set threshold;

[0179] A setting module, used for setting all agents of the same category as a category node according to the categories of the multiple agents, each category node corresponding to a category;

[0180] The second building module is used to build a category edge between two category nodes.

[0181] Optionally, the updating module 13 includes:

[0182] A first calculation module, used to calculate the agent-level interaction attention scores between the spatiotemporal nodes of the spatiotemporal interaction graph;

[0183] A Laplace normalized matrix acquisition module is used to establish an adjacency matrix according to the space-time edges of the space-time interaction graph, perform Laplace transform, and obtain a Laplace normalized matrix;

[0184] An agent-level interaction matrix acquisition module, used for multiplying the agent-level interaction attention score with the Laplace normalization matrix to obtain an agent-level interaction matrix;

[0185] A second calculation module is used to calculate the category-level interaction attention scores between the category nodes of the spatiotemporal interaction graph, and obtain the category-level interaction attention after normalization, which is also used as the category edge representing the interaction relationship between the category nodes;

[0186] A global attention acquisition module, used to obtain global attention by performing an asymmetric convolution operation on the agent-level interaction matrix and then performing nonlinear activation;

[0187] A fusion module, used for fusing the category-level interaction attention with the global-level attention to obtain a final attention of the spatiotemporal interaction graph, and using the final attention as an adjacency matrix of the spatiotemporal interaction graph;

[0188] The interactive association relationship model based on the graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to achieve the state update of the node itself.

[0189] Optionally, the multimodal trajectory prediction module 14 includes:

[0190] A mapping module, used for mapping the features output by the interactive association relationship model into latent variables through the learnable mapping based sampling method;

[0191] A decoding module is used to decode the latent variable into K future trajectories of the agent through the conditional variational autoencoder.

[0192] Based on the same inventive concept, another embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the interactive correlation relationship model and ship trajectory prediction method based on the graph neural network as described in any of the above embodiments.

[0193] Based on the same inventive concept, another embodiment of the present invention further provides a computer program product, including a computer program, which is executed by a processor as the interactive correlation relationship model and ship trajectory prediction method based on graph neural network as described in any of the above embodiments.

[0194] Based on the same inventive concept, another embodiment of the present invention further provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the interactive correlation relationship model and ship trajectory prediction method based on the graph neural network as described in any of the above embodiments is implemented.

[0195] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0198] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0203] The above is a detailed introduction to the interactive correlation relationship model and ship trajectory prediction method based on graph neural network provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An interactive correlation model and ship trajectory prediction method based on graph neural network, characterized in that: The method comprises: The historical trajectory information of different types of agents is encoded through the multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents; According to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, a spatiotemporal interaction graph is constructed, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories; Through the interactive association relationship model based on the graph neural network, the interactive attention mechanism at the agent level, category level and global level is constructed, the interactive association relationship is integrated into its own state update, and the modeling ability of the interactive relationship in the multi-agent scenario is enhanced, including: calculating the agent-level interactive attention score between the spatiotemporal nodes of the spatiotemporal interaction graph; establishing an adjacency matrix according to the spatiotemporal edges of the spatiotemporal interaction graph, performing Laplace transform, and obtaining a Laplace normalized matrix; multiplying the agent-level interactive attention score with the Laplace normalized matrix to obtain an agent-level interaction matrix; calculating the agent-level interactive attention score between the category nodes of the spatiotemporal interaction graph; The category-level interaction attention scores between the nodes are normalized to obtain the category-level interaction attention, which is also used as the category edge representing the interaction relationship between category nodes; the global-level attention is obtained by performing an asymmetric convolution operation on the agent-level interaction matrix and then performing a nonlinear activation; the final attention of the spatiotemporal interaction graph is obtained by fusing the category-level interaction attention with the global-level attention, and the final attention is used as the adjacency matrix of the spatiotemporal interaction graph; the interaction association relationship model based on the graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to realize the state update of the node itself; Through the multimodal trajectory prediction module based on the learnable mapping sampling method and the conditional variational autoencoder, the multimodal distribution of the agent's future trajectory is captured, and multiple possible future trajectories of each agent are obtained; The historical trajectory information of M agents detected from the historical frame is encoded, and the current trajectory information of N agents detected from the current frame is encoded to obtain the semantic features of M+N agents, including: inputting the historical trajectory information of the M agents contained in the historical frame into a plurality of long short-term memory artificial neural networks with different parameter structures, and outputting the semantic features of the M agents through the plurality of long short-term memory artificial neural networks with different parameter structures; inputting the current trajectory information of the N agents contained in the current frame into a plurality of multilayer perceptrons with different parameter structures, and outputting the obtained semantic features of the N agents through the plurality of multilayer perceptrons with different parameter structures; According to the semantic features of the M+N agents, whether there is an interactive relationship between the M agents in the historical frame, whether there is an interactive relationship between the N agents in the current frame, and whether the M agents in the historical frame can match the N agents in the current frame, a graph structure with M+N nodes is constructed, each node in the graph structure corresponds to the semantic feature of an agent, and each edge in the graph structure represents the existence of an interactive relationship between two agents, including: according to the trajectory information of the M agents detected in the historical frame and the position information of the N agents detected in the current frame, M+N nodes are created, each node corresponds to the semantic feature of an agent; according to the semantic features of the M+N agents, whether the distance between each two agents in the three-dimensional space is less than a set threshold; when the distance between the two agents in the three-dimensional space is less than the set threshold, an edge is constructed between the nodes corresponding to the two agents; Construct an interactive association relationship model based on an L-layer graph neural network, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in current frames, and update the semantic features of the M+N agents through the interactive association relationship model to obtain updated semantic features of the agents, and obtain updated edge features based on the updated semantic features; The historical trajectory information of the M agents detected in the historical frame, the updated semantic features of the M agents and the updated edge features are input into a multimodal trajectory prediction module based on a conditional variational autoencoder to obtain multiple future trajectory information of the M agents.

2. The interactive correlation relationship model and ship trajectory prediction method based on graph neural network according to claim 1 is characterized in that: The multi-type agent encoding module encodes the historical trajectory information of different types of agents respectively to obtain the spatiotemporal characteristics of multiple agents, including: In view of the fact that the motion behavior patterns of different types of intelligent agents are different, the trajectory data of intelligent agents with different motion patterns are encoded in a differentiated manner, including: Standardize the spatiotemporal features of different dimensions of different categories of agents; Different LSTM encoders are used to extract the spatiotemporal features of different types of agents through different network structure parameters. The structure parameters of the same type of agents are the same, while the structure parameters of different types of agents are different.

3. The interactive correlation model and ship trajectory prediction method based on graph neural network according to claim 1 is characterized in that: The step of constructing a spatiotemporal interaction graph according to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents includes: According to the multiple intelligent agents, create multiple spatiotemporal nodes corresponding to the number of the multiple intelligent agents, each spatiotemporal node corresponding to one intelligent agent; According to the spatial positions of the multiple agents, determining whether the distance between each two agents in the three-dimensional space is less than a set threshold; When the distance between two agents in the three-dimensional space is less than the set threshold, a spatiotemporal edge is constructed between the nodes corresponding to the two agents; According to the categories of the multiple agents, all agents of the same category are set as a category node, each category node corresponds to a category; Build a category edge between two category nodes.

4. The interactive correlation relationship model and ship trajectory prediction method based on graph neural network according to claim 1 is characterized in that: The multimodal trajectory prediction module based on the learnable mapping sampling method and the conditional variational autoencoder is used to capture the multimodal distribution of the agent's future trajectory and obtain multiple possible future trajectories of each agent, including: Mapping the features output by the interactive correlation model into latent variables through the learnable mapping-based sampling method; The latent variables are decoded into K future trajectories of the agent through the conditional variational autoencoder.

5. The interactive correlation model and ship trajectory prediction method based on graph neural network according to claim 1 is characterized in that: In the process of constructing the interactive attention mechanism at the agent level, category level and global level based on the interactive association relationship model of the graph neural network, the attention construction formula is: ； ； ； ； in, , Representative Category , The characteristics of || are concatenation operations. is the learnable attention weight vector, is a nonlinear activation function, is the maximum pooling operation, Representative Category With category The attention score between represents the category-level attention after weight distribution, is the global attention obtained from the agent-level attention, Represents the ultimate attention, The output of the interactive relationship model is Representation of layer interaction relationships.

6. An interactive correlation model and ship trajectory prediction device based on graph neural network, characterized in that: The device comprises: An encoding module, used to encode the historical trajectory information of different types of agents respectively through a multi-type agent encoding module to obtain the spatiotemporal characteristics of multiple agents; A spatiotemporal interaction graph construction module, for constructing a spatiotemporal interaction graph according to the spatiotemporal characteristics of the multiple agents, whether there is an interactive association relationship between the multiple agents, and the categories of the multiple agents, wherein the spatiotemporal nodes in the spatiotemporal interaction graph correspond to the observed agents, the spatiotemporal edges in the spatiotemporal interaction graph represent the interactive association relationship between the agents, the category nodes in the spatiotemporal interaction graph correspond to the observed agents belonging to the same category, and the category edges in the spatiotemporal interaction graph represent the interactive association relationship between agents of multiple categories; The spatiotemporal interactive attention calculation and update module is used to build interactive attention mechanisms at the agent level, category level, and global level through an interactive correlation model based on a graph neural network, integrate interactive correlations into its own state updates, and enhance the modeling capabilities of interactive relationships in multi-agent scenarios; A multimodal trajectory prediction module is used to capture the multimodal distribution of the agent's future trajectory through a multimodal trajectory prediction module based on a learnable mapping sampling method and a conditional variational autoencoder, and obtain multiple possible future trajectories of each agent; A semantic feature enhancement module is used to encode the historical trajectory information of M agents detected from the historical frame and the current trajectory information of N agents detected from the current frame to obtain the semantic features of M+N agents, including: inputting the historical trajectory information of the M agents contained in the historical frame into a plurality of long short-term memory artificial neural networks with different parameter structures, and outputting the semantic features of the M agents through the long short-term memory artificial neural networks with different parameter structures; inputting the current trajectory information of the N agents contained in the current frame into a plurality of multilayer perceptrons with different parameter structures, and outputting the obtained semantic features of the N agents through the multilayer perceptrons with different parameter structures; The first processing module is used to construct a graph structure with M+N nodes according to the semantic features of the M+N agents, whether there is an interactive relationship between the M agents in the historical frame, whether there is an interactive relationship between the N agents in the current frame, and whether the M agents in the historical frame can match the N agents in the current frame, wherein each node in the graph structure corresponds to the semantic feature of an agent, and each edge in the graph structure represents the existence of an interactive relationship between two agents, including: creating M+N nodes according to the trajectory information of the M agents detected in the historical frame and the position information of the N agents detected in the current frame, wherein each node corresponds to the semantic feature of an agent; judging whether the distance between each two agents in the three-dimensional space is less than a set threshold according to the semantic features of the M+N agents; and constructing an edge between the nodes corresponding to the two agents when the distance between the two agents in the three-dimensional space is less than the set threshold; The second processing module is used to construct an interactive association relationship model based on an L-layer graph neural network, wherein each layer includes nodes detected in M ​​historical frames and N nodes detected in current frames, and the semantic features of the M+N agents are updated through the interactive association relationship model to obtain updated semantic features of the agents, and updated edge features are obtained according to the updated semantic features; A third processing module is used to input the historical trajectory information of the M agents detected in the historical frame, the updated semantic features of the M agents, and the updated edge features into a multimodal trajectory prediction module based on a conditional variational autoencoder to obtain multiple future trajectory information of the M agents; The spatiotemporal interactive attention calculation update module includes: A first calculation module, used to calculate the agent-level interaction attention scores between the spatiotemporal nodes of the spatiotemporal interaction graph; A Laplace normalized matrix acquisition module is used to establish an adjacency matrix according to the space-time edges of the space-time interaction graph, perform Laplace transform, and obtain a Laplace normalized matrix; An agent-level interaction matrix acquisition module, used for multiplying the agent-level interaction attention score with the Laplace normalization matrix to obtain an agent-level interaction matrix; A second calculation module is used to calculate the category-level interaction attention scores between the category nodes of the spatiotemporal interaction graph, and obtain the category-level interaction attention after normalization, which is also used as the category edge representing the interaction relationship between the category nodes; A global attention acquisition module, used to obtain global attention by performing an asymmetric convolution operation on the agent-level interaction matrix and then performing nonlinear activation; A fusion module, used for fusing the category-level interaction attention with the global-level attention to obtain a final attention of the spatiotemporal interaction graph, and using the final attention as an adjacency matrix of the spatiotemporal interaction graph; The interactive association relationship model based on the graph neural network includes L layers, which extracts and models multi-agent interactions through graph convolution operations to achieve the state update of the node itself.

7. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the interactive correlation relationship model and ship trajectory prediction method based on graph neural network as described in any one of claims 1 to 5.

8. A computer-readable medium, characterized in that A computer program is stored thereon, wherein when the computer program is executed by a processor, the interactive correlation relationship model and ship trajectory prediction method based on a graph neural network as described in any one of claims 1 to 5 is implemented.

Images