A multi-modal trajectory prediction method based on a conditional diffusion model

Abstract

The application discloses a kind of multi-modal trajectory prediction methods based on conditional diffusion model, belong to trajectory prediction technical field.The method based on the conditional diffusion model framework of non-self-recurrence of the present application, the trajectory prediction task is conceptualized as the guided inverse process of motion uncertainty diffusion, by gradually eliminating the uncertainty of marine area, definite simulation ship movement is transformed from uncertainty to deterministic state;Utilize the multi-modal mixing of interactive information and physical information as generation condition to guide each stage of reverse diffusion process, to capture the inherent complexity and uncertainty in maritime behavior, and then improve the trajectory prediction performance.In addition, the trade-off between trajectory sample diversity and certainty can be achieved by adjusting the length of Markov chain.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction technology, and more specifically to a multimodal trajectory prediction method based on a conditional diffusion model. Background Technology

[0002] Trajectory prediction is crucial for maritime safety systems, aiding in functions such as maritime traffic monitoring, collision avoidance, anomaly detection, and threat assessment. However, developing accurate prediction models is extremely challenging due to the complexity of human behavior and the various factors influencing ship motion. Traditional physics-based models have proven effective for short-term predictions, but struggle to achieve long-term accuracy due to their idealized assumptions. To address this limitation, the shift to Long Short-Term Memory (LSTM) networks and related variants has become a representative approach for ship trajectory prediction. However, ship trajectory prediction is inherently probabilistic and multimodal, highlighting the importance of accurately predicting unbiased distributions encompassing a variety of potential future paths. Existing research utilizes deep generative models (GANs, CVAEs) to traverse probability distributions across all possible future trajectories. However, these models leverage latent variables rather than explicitly modeling the inherent uncertainties in the data distribution, thus introducing limitations in capturing the full complexity of ship motion.

[0003] Furthermore, ship motion is significantly influenced by the surrounding environment, including physical constraints and different motion patterns in different geographical regions. In this regard, it is necessary to incorporate physical priors into the modeling process to avoid potential collision risks. In addition, spatial interactions between adjacent vessels should also be considered. Even if a ship has a relatively definite long-term goal, its actual motion is affected by the complex dynamics of interactions between different vessels, especially during close encounters. Therefore, this patent proposes a multimodal trajectory prediction method based on a conditional diffusion model. Based on a non-autoregressive diffusion model framework and conditional modeling, it significantly improves the performance of ship trajectory prediction. This integration is crucial for developing realistic marine scenarios that accurately reflect the different conditions and navigation behaviors encountered in different marine environments. Summary of the Invention

[0004] The purpose of this invention is to address the limitations of existing deep generative models in capturing the full complexity of ship motion due to their failure to explicitly model the inherent uncertainties in data distribution, and the lack of consideration for the significant impact of physical constraints and spatial interactions on ship trajectories. Therefore, this invention, based on a conditional diffusion framework, utilizes a mixture of interactive and physical information as guiding factors for trajectory generation, directing each stage of the backdiffusion process. This includes leveraging past trajectories of the target ship and surrounding vessels, as well as scene context, to provide a diverse and plausible range of future trajectories. Compared to other latent variable-based deep generative models, this invention explicitly simulates the evolution of ship motion from an uncertain state to a deterministic state, while capturing the inherent complexity and uncertainty of maritime behavior, thus resolving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention provides a multimodal trajectory prediction method based on a conditional diffusion model, comprising the following steps:

[0007] Step S10: Determine the future trajectory of the target vessel Forward diffusion is performed, gradually adding Gaussian noise to transform the trajectory into a white Gaussian noise vector. Where k is the number of diffusion steps;

[0008] Step S20: Construct a conditional encoder ψ to generate conditional embeddings that guide the backdiffusion step;

[0009] Step S201: Transfer the target ship's observation history x -L+1：0 Encoding as high-dimensional conditional features Z en ;

[0010] Step S202: Model the concatenated features of the ship itself and its neighbors The high-dimensional encoded representation Z of spatial interaction information is obtained. neigh :

[0011] Step S203: Create pixel-level alignment by encoding physical scene information into high-dimensional feature Z using the scene context image and observed 2D coordinate trajectory. map ;

[0012] Step S30: The mixture of the conditional encoding representations obtained in steps S201, S202, and S203 is used as the condition guiding the back diffusion learning process, expressed as:

[0013] Step S40: Define the back-diffusion denoising network θ, using the diffused future trajectory. The denoising step number k and the conditional embedding c are used as inputs to generate the denoised trajectory;

[0014] Step S401: Calculate the temporal embedding k emb =[β k sin(β) k ), cos(β) k )] to encode the time dimension information of the diffusion process, where β k From fixed sample values;

[0015] Step S402: The features are explicitly upsampled through multiple fully connected layers and element-wise multiplication to obtain the feature aggregation representation;

[0016] Step S403: Use a decoder to capture complex spatiotemporal dependencies and downsample the output sequence to the trajectory dimension to form a sequence with... Noise prediction in the same dimension

[0017] Step S50: During the training phase, define the loss function as the minimum mean squared error loss to update the model parameters;

[0018] Step S60: In the inference phase, the non-Markovian denoising strategy DDIM is adopted. By reconstructing the original data from the noisy state and then deterministically estimating the direction of the next target timestamp, a denoising trajectory is obtained by iteratively applying denoising steps that reduce timestamps.

[0019] In the above scheme, the specific steps of step S10 are as follows:

[0020] Step S101, the forward diffusion process stage, gradually introduces Gaussian noise into the clean future trajectory sequence. In the process, it is transformed into a white Gaussian noise vector through k diffusion steps. The process is represented as follows:

[0021]

[0022] in, Let represent the parameters of the k-th diffusion step, and ∈ represent Gaussian noise.

[0023] In the above scheme, the specific steps of step S20 are as follows:

[0024] Step S201, the historical information encoder will record the target ship's observation history x -L+1：0 The encoding is performed by using an LSTM network to capture the temporal dependencies between all states and encoding them as high-dimensional features, as shown below:

[0025] Z en =LSTMen (x -L+1：0 )

[0026] x -L+1：0 =[x -L+1 ,x -L+1 [x0]

[0027] Where x -L+1：0 This represents the historical observation trajectory of the target vessel, indicating the trajectory from x. -L+1 LSTM for all historical trajectory points up to x0 en Represents the Long Short-Term Memory network;

[0028] Step S202, the task of the spatial interaction information encoder is to model the trajectory information of neighboring ships around the ship. First, the scene at each timestamp t is abstracted into a spatiotemporal directed graph G = (V, E) to model the neighbor information of each ship, where nodes represent ships and edges represent their spatial interactions. The Euclidean distance l2 is calculated as a factor in whether ships influence each other, where the threshold of the influence distance is set as a hyperparameter.

[0029] l2=||(x i -x j ) 2 +(y i -y j ) 2 ||

[0030] Where (x) i x j ), (x j y j ) represent the coordinates of the two points, and ||*|| represents the Euclidean norm of the vector, i.e., the length of the vector;

[0031] Since the number of neighbors for each ship changes over time, it is necessary to aggregate their past trajectories, given a set of neighboring ship trajectories. in Represents the i-th trajectory, containing the trajectory from x. -L+1 L historical trajectory points up to x0, where i represents the trajectory index from 1 to N, L represents the backtracking time step, and N is the total number of trajectories in the set, and adjacent ship trajectory sets. The aggregation state is calculated as follows:

[0032]

[0033] In this approach, element-wise addition is used to preserve the fixed structure of the encoder input and counting information, while processing the concatenated features of itself and its neighbors to obtain a richer representation.

[0034]

[0035] Step S203, to create pixel-level alignment between different modalities to learn a rich semantic context representation that effectively utilizes scene context (similar to an image) and observation trajectory (2D coordinates), firstly, a Gaussian kernel is used to align past trajectories x -L+1：0 =[x -L+1 x -L+1 The trajectory heatmap H is transformed from [x0, ..., x0] and calculated as follows:

[0036]

[0037] Where σ is the standard deviation of the Gaussian kernel, (x t y t The coordinates (t) represent the longitude and latitude coordinates at timestamp t. If two Gaussian coordinates overlap, element-wise summation is used to ensure the strength and smoothness of the trajectory heatmap. 2D-CNN is used to extract joint features from the scene context and past trajectories, and the trajectory heatmap is fused with the image to represent:

[0038]

[0039] Where α is a hyperparameter, It is a scene image, ⊙ represents element-wise multiplication, and H is a trajectory heatmap.

[0040] In the above scheme, the specific steps of step S30 are as follows:

[0041] In step S301, the conditional aggregator concatenates and transforms the features obtained from parts S201, S202, and S203 through the MLP layer, as shown below:

[0042]

[0043] In the above scheme, the specific steps of step S40 are as follows:

[0044] Step S401, essentially, the backdiffusion network is a noise prediction network that uses diffused future trajectories. Taking the denoising step k and the conditional embedding c as input, the diffusion trajectories obtained at different time steps are first feature-encoded by calculating the temporal embedding representation, as shown below:

[0045] k emb =[β k sin(β) k ), cos(β) k )]

[0046] Where, β k From a fixed sample, concatenate with a condition c along the variable dimension to generate a new condition c. k ,enter The future trajectory embedding of the diffusion is obtained by transformation through linear projection.

[0047] Step S402: Apply several fully connected layers and use element-wise multiplication to explicitly embed the input in the feature dimension. and new condition c k Upsampling is performed to obtain an aggregated representation of the features.

[0048] ω=sigmoid(FC1(c k ))

[0049] b = FC2(c k )

[0050]

[0051] Where FC1, FC2, and FC3 represent fully connected layers, and sigmoid is the activation function. Broadcast via timestamp to form right Broadcasting with the same timestamp

[0052] In step S403, before the decoder layer, a position embedding is introduced in the implementation to inject information about the relative positions of different timestamps. Subsequently, the Transformer-based decoder consists of multiple self-attention layers to model the complex spatiotemporal dependencies of the fused features. The self-attention modules facilitate the enhanced interactive sequence representation between timestamps by utilizing derived queries, keys, and values, as shown below:

[0053]

[0054] Where SelfAttention represents the self-attention module, and PositionEncoding represents the position encoding module.

[0055]

[0056] Where FFN represents a feedforward neural network, which uses several fully connected layers with the same aggregation operation to process the output sequence e. k Downsampling to the trajectory dimension to form with Noise prediction in the same dimension The denoised output is calculated deterministically and is expressed as:

[0057]

[0058] in The denoised trajectory represents the estimated trajectory at step ks, and ∈θ(*) represents the noise prediction function, which is based on the current diffusion trajectory. The denoising step k and conditional embedding c are used to predict the noise component in the trajectory. These are hyperparameters obtained from fixed sampling, and s is the step size reduction used to accelerate the back diffusion process. This is an estimate of the original trajectory state at step k, calculated as follows:

[0059]

[0060] in These are hyperparameters obtained from fixed sampling. It is the denoised trajectory, representing the trajectory estimation at denoising step k.

[0061] In the above scheme, the specific steps of step S50 are as follows:

[0062] Step S501: The training loss function is defined as the mean squared error loss. Both diffusion and backdiffusion are modeled as Markov chains with Gaussian transitions. Trajectory prediction is performed by modeling the joint distribution, as shown in the following model:

[0063]

[0064] in, This represents the trajectory from the initial Gaussian noise under given condition c. To the final trajectory The joint probability distribution, Represents the initial Gaussian noise trajectory The probability distribution of p θ Represents a conditional probability distribution. Indicates from arrive The probability distribution of each intermediate trajectory, where K is the total number of steps in the forward diffusion process. The initial trajectory is indicated to follow a Gaussian distribution with a mean of 0 and a covariance of the identity matrix I. c is the conditional embedding obtained in step S301, which includes the observed trajectories of the target vessel itself and neighboring vessels, as well as conditional information such as multimodal scene context. Simultaneously:

[0065]

[0066] in, This indicates that the conditional probability follows a certain pattern. The mean, The variance follows a normal distribution. This represents the mean estimation output controlled by the neural network parameters θ, depending on the denoising step k, under given condition c. This represents the variance obtained by using fixed sample values.

[0067] Since both the forward diffusion and backward diffusion processes are Gaussian distributions, the training objective is naturally transformed into calculating the difference between the means:

[0068]

[0069] in This indicates that the process follows a reverse diffusion, and given c, it represents the approximate posterior mean estimate at denoising step k. This represents the mean estimate given by the noise prediction network with parameter θ at the denoising step k, given condition c.

[0070] Finally, a conditional denoiser is learned by applying the following simplified loss function at each step k, which yields better generative properties, as shown below:

[0071]

[0072] 7. The multimodal trajectory prediction research method based on the conditional diffusion model according to claim 1, characterized in that: the specific steps of step S60 are as follows:

[0073] Step S601: The backdiffusion process employs the non-Markovian denoising strategy DDIM. This strategy initiates by reconstructing the original data from the noisy state and then deterministically estimates the direction to the next target timestamp. Once the encoder ψ and decoder θ are trained, a reasonable trajectory can be generated through the denoising step. Notably, the initial input is Gaussian noise. The iteration count is K times per second. A denoising trajectory is obtained by iteratively applying denoising steps that reduce timestamps. This will serve as the final prediction.

[0074] The beneficial effects of this invention are:

[0075] 1. This invention, by conceptualizing the trajectory prediction task as a guided inverse process of motion pattern uncertainty diffusion, explicitly simulates the transformation of motion from uncertainty to determinism, solves the problem that existing deep generative models cannot explicitly model the inherent uncertainty in data distribution, and achieves the effect of improving the model's ability to fully capture the dynamics and complexity of trajectory motion, effectively handling future uncertainties in dynamic scenarios.

[0076] 2. This invention uses multimodal fusion of external condition information such as neighboring spatial interactions and physical constraints (steps 20 and 30) to encode the historical trajectories of the target ship and neighboring ships into state embeddings. Combined with the observed trajectory and the static scene context to form a two-dimensional pixel-level heat map, this invention ensures the effective integration of physical prior information in the modeling process. This solves the problem that existing technologies have difficulty modeling surrounding scene environmental factors, and achieves the effect of generating spatially sensitive and physically reasonable future trajectories, significantly improving the accuracy of prediction.

[0077] 3. The present invention uses a non-autoregressive trajectory generation method (step 60) to sample noise trajectories from a predefined Gaussian distribution and gradually derive accurate future trajectories using a backdiffusion process. This solves the problems of error accumulation and weak resistance to noise data in existing methods, and achieves the effect of reducing prediction errors and improving model robustness. Attached Figure Description

[0078] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Please see Figure 1 The present invention provides a technical solution:

[0081] This invention provides the following technical solution: a multimodal trajectory prediction method based on a conditional diffusion model. Specific steps are illustrated using ship trajectory prediction in the Denmark Strait region as an example. The specific steps for applying the trajectory prediction method to ship trajectory prediction in the Denmark Strait region are as follows:

[0082] Step 1: Predict the future trajectory of the target vessel Forward diffusion is performed, gradually adding Gaussian noise to transform the trajectory into a white Gaussian noise vector. Where k is the number of diffusion steps;

[0083] Step 2: Construct a conditional encoder ψ to generate conditional embeddings that guide the backdiffusion step;

[0084] Step 2-1: Use the historical information encoder to record the target ship's observation history x -L+1：0The encoding is performed by using an LSTM network to capture the temporal dependencies between all states and encoding them as high-dimensional features, as shown below:

[0085] Z en =LSTM en (x -L+1：0 )

[0086] x -L+1：0 =[x -L+1 ,x -L+1 [x0]

[0087] Where x -L+1：0 This represents the historical observation trajectory of the target vessel, indicating the trajectory from x. -L+1 LSTM for all historical trajectory points up to x0 en Represents the Long Short-Term Memory network;

[0088] Step 2-2: The task of the spatial interaction information encoder is to model the trajectory information of neighboring ships around a vessel. First, the scene at each timestamp t is abstracted into a spatiotemporal directed graph G = (V, E) to model the neighbor information of each ship, where nodes represent ships and edges represent their spatial interactions. The Euclidean distance l² is calculated as a factor in whether ships influence each other, with the threshold of the influence distance set as a hyperparameter.

[0089] l2=||(x i -x j ) 2 +(y i -y j ) 2 ||

[0090] Where (x) i x j ), (x j y j ) represent the coordinates of the two points, and ||*|| represents the Euclidean norm of the vector, i.e., the length of the vector;

[0091] Since the number of neighbors for each ship changes over time, it is necessary to aggregate their past trajectories, given a set of neighboring ship trajectories. in Represents the i-th trajectory, containing the trajectory from x. -L+1 L historical trajectory points up to x0, where i represents the trajectory index from 1 to N, L represents the backtracking time step, and N is the total number of trajectories in the set, and adjacent ship trajectory sets. The aggregation state is calculated as follows:

[0092]

[0093] In this approach, element-wise addition is used to preserve the fixed structure of the encoder input and counting information, while processing the concatenated features of itself and its neighbors to obtain a richer representation.

[0094]

[0095] Steps 2-3 involve modeling scene context information. Our implementation involves creating pixel-level alignment between different modalities to learn a rich semantic context representation that effectively utilizes scene context (similar to an image) and view trajectory (2D coordinates). First, a Gaussian kernel is used to model the past trajectory x... -L+1：0 =[x -L+1 x -L+1 The trajectory heatmap H is transformed from [x0, ..., x0] and calculated as follows:

[0096]

[0097] Where σ is the standard deviation of the Gaussian kernel, (x t，yt The coordinates (t) represent the longitude and latitude coordinates at timestamp t. If two Gaussian coordinates overlap, element-wise summation is used to ensure the strength and smoothness of the trajectory heatmap. 2D-CNN is used to extract joint features from the scene context and past trajectories, and the trajectory heatmap is fused with the image to represent:

[0098]

[0099] Where α is a hyperparameter. It is a scene image, ⊙ represents element-wise multiplication, and H is a trajectory heatmap.

[0100] Step 3: The mixture of the conditional encoding representations obtained in steps S201, S202, and S203 is used as the condition to guide the back-diffusion learning process. The features are concatenated and transformed by the conditional aggregator MLP, as shown below:

[0101]

[0102] Step 4: Define the back-diffusion denoising network θ, using the diffused future trajectory. Denoising steps k and conditional embedding c As input, a denoised trajectory is generated;

[0103] Step 4-1: By calculating the temporal embedding representation, the diffusion trajectories obtained at different time steps are feature-encoded, as shown below:

[0104] k emb =[β k sin(β) k ), cos(β) k )]

[0105] Where, β k From a fixed sample, concatenate with a condition c along the variable dimension to generate a new condition c. k ,enter The future trajectory embedding of the diffusion is obtained by transformation through linear projection.

[0106] Step 4-2: Apply several fully connected layers and use element-wise multiplication to explicitly embed the input into the feature dimension. and new condition c k Upsampling is performed to obtain an aggregated representation of the features.

[0107] ω=sigmoid(FC1(c k ))

[0108] b = FC2(c k )

[0109]

[0110] Where FC1, FC2, and FC3 represent fully connected layers, and sigmoid is the activation function. Broadcast via timestamp to form right Broadcasting with the same timestamp

[0111] Step 4-3 introduces a location embedding before the decoder layer to inject information about the relative positions of different timestamps. Subsequently, the Transformer-based decoder consists of multiple self-attention layers to model the complex spatiotemporal dependencies of the fused features. The self-attention modules facilitate enhanced interactive sequence representations between timestamps by leveraging derived queries, keys, and values, as shown below:

[0112]

[0113] Where SelfAttention represents the self-attention module, and PositionEncoding represents the position encoding module.

[0114]

[0115] Where FFN represents a feedforward neural network, which uses several fully connected layers with the same aggregation operation to process the output sequence e. k Downsampling to the trajectory dimension to form with Noise prediction in the same dimension The denoised output is calculated deterministically and is expressed as:

[0116]

[0117] in The denoised trajectory represents the estimated trajectory at step ks, ∈ θ (*) represents the noise prediction function, which is based on the current diffusion trajectory. The denoising step k and conditional embedding c are used to predict the noise component in the trajectory. These are hyperparameters obtained from fixed sampling, and s is the step size reduction used to accelerate the back diffusion process. This is an estimate of the original trajectory state at step k, calculated as follows:

[0118]

[0119] in These are hyperparameters obtained from fixed sampling. It is the denoised trajectory, representing the trajectory estimation at denoising step k.

[0120] Step 5: During the training phase, update the model parameters by defining the loss function as mean squared error loss;

[0121] Step 5-1: The training loss function is defined as the mean squared error loss. Specifically, both diffusion and backdiffusion are modeled as Markov chains with Gaussian transitions. Trajectory prediction is performed by modeling the joint distribution, as shown in the following modeling representation:

[0122]

[0123] in, This represents the trajectory from the initial Gaussian noise under given condition c. To the final trajectory The joint probability distribution, Represents the initial Gaussian noise trajectory The probability distribution of p θ Represents a conditional probability distribution. Indicates from arrive The probability distribution of each intermediate trajectory, where K is the total number of steps in the forward diffusion process. The initial trajectory is indicated to follow a Gaussian distribution with a mean of 0 and a covariance of the identity matrix I. c is the conditional embedding obtained in step S301, which includes the observed trajectories of the target vessel itself and neighboring vessels, as well as conditional information such as multimodal scene context. Simultaneously:

[0124]

[0125] in, This indicates that the conditional probability follows a certain pattern. The mean, The variance follows a normal distribution. This represents the mean estimation output controlled by the neural network parameters θ, depending on the denoising step k, under given condition c. This represents the variance obtained by using fixed sample values.

[0126] Since both the forward diffusion and backward diffusion processes are Gaussian distributions, the training objective is naturally transformed into calculating the difference between the means:

[0127]

[0128] in This indicates that the process follows a reverse diffusion, and given c, it represents the approximate posterior mean estimate at denoising step k. This represents the mean estimate given by the noise prediction network with parameter θ at the denoising step k, given condition c.

[0129] Finally, a conditional denoiser is learned by applying the following simplified loss function at each step k, which yields better generative properties, as shown below:

[0130]

[0131] Step 6: In the inference phase, the non-Markovian denoising strategy DDIM is adopted. This involves reconstructing the original data from the noisy state, then deterministically estimating the direction of the next target timestamp, and iteratively applying denoising steps that reduce timestamps to obtain the denoised trajectory.

[0132] Step 6-1, during the inference phase, the backdiffusion process employs the non-Markovian denoising strategy DDIM. This strategy initiates by reconstructing the original data from the noisy state and then deterministically estimates the direction to the next target timestamp. Once the encoder ψ and decoder θ are trained, a reasonable trajectory can be generated through the denoising step. Notably, the initial input is Gaussian noise. The iteration count is K times per second. A denoising trajectory is obtained by iteratively applying denoising steps that reduce timestamps. As a final prediction.

[0133] The research method for multimodal trajectory prediction based on the conditional diffusion model includes a forward diffusion module, a conditional encoder module ψ, and a backward diffusion denoising network θ, specifically:

[0134] During the forward diffusion phase, Gaussian noise is gradually added to the clean future trajectory. In the process, it is transformed into a white Gaussian noise vector in k diffusion steps.

[0135] In the conditional encoder module, the mixture of various conditional encoded representations is used as a condition to guide the back diffusion learning process, and the features are concatenated and transformed through MLP.

[0136] In the back-diffusion denoising network θ, the diffused future trajectory is used. The denoising step k and the conditional embedding c are used as inputs to iteratively generate the denoising trajectory;

[0137] The method of this invention takes trajectory prediction as its research object. Based on a non-autoregressive diffusion model framework, it conceptualizes the trajectory prediction task as a guided inverse process of motion uncertainty diffusion. By gradually eliminating the uncertainty in the ocean region, it clearly simulates the transition of ship motion from uncertainty to determinism. It utilizes the multimodal mixture of interactive and physical information as generation conditions to guide each stage of the inverse diffusion process, capturing the inherent complexity and uncertainty in maritime behavior, thereby improving trajectory prediction performance. Furthermore, the length of the Markov chain can be adjusted to achieve a trade-off between the diversity and determinism of generated trajectory samples. Although the invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-modal trajectory prediction method based on a conditional diffusion model, characterized in that, The method comprises the following steps: Step S10: future trajectory of the target ship forward diffusion is performed to convert the trajectory into a white Gaussian noise vector by adding Gaussian noise step by step wherein is the number of diffusion steps; Step S20: constructing a condition encoder generating a condition embedding that guides the back diffusion step; Step S201: Obtain target ship observation history Encoding into high-dimensional conditional features ; Step S202: modelling the concatenation of features of the ship itself and of the neighbours , obtaining a high-dimensional encoded representation of the spatial interaction information ; Step S203: Create pixel-level alignment, encode physical scene information into high-dimensional features with scene context image and observed 2D coordinate trajectory ; Step S30: mixing the respective condition encoding representations obtained in steps S201, S202 and S203 as a condition to guide the process of backpropagation learning, denoted as ; Step S40: defining a reverse diffusion denoising network that takes the diffused future trajectory , the denoising steps , and the conditional embedding as input, thereby generating a denoised trajectory with the same number of denoising steps as diffusion steps, ; Step S401: Calculate time embedding In the encoding diffusion process time dimension information, wherein From the fixed sampling value; Step S402: The features are up-sampled by a plurality of fully connected layers and element-wise multiplication to obtain a feature aggregation representation; Step S403: Use a decoder to capture complex spatiotemporal dependencies and downsample the output sequence to the trajectory dimension to form a sequence with... Noise prediction in the same dimension Step S50: During the training phase, define the loss function as the minimum mean squared error loss to update the model parameters; Step S60: Inference phase, using a non-Markov denoising strategy DDIM, by reconstructing the original data from the noisy state, then deterministically estimating the direction to the next target timestamp, iteratively applying the denoising step with reduced number of timestamps to obtain the denoised trajectory .

2. The method of claim 1, wherein: The specific steps of the step S10 are as follows: Step S101, forward diffusion process phase, gradually introduce Gaussian noise into the clean future trajectory sequence In the middle, by The number of diffusion steps is converted into a white Gaussian noise vector The process is represented as follows: wherein, represents a parameter of the diffusion step, represents a Gaussian noise.

3. The method of claim 1, wherein: The specific steps of the step S20 are as follows: Step S201, the history information encoder encodes the target ship observation history Encoding is performed, specifically using LSTM networks to capture the temporal dependencies between all states and encode them into high-dimensional features, represented as follows: wherein represents an observed historical trajectory of the target ship, represents all historical trajectory points from to , represents a long short-term memory network; Step S202, the task of the spatial interaction information encoder is to model the trajectory information of the neighbor ships around the ship, first, the scene of each timestamp is abstracted as a spatio-temporal directed graph to model the neighbor information of each ship, where the node represents the ship and the edge represents their spatial interaction, by calculating the Euclidean distance as a factor of whether the ships affect each other, where the threshold of the influence distance is set as a hyperparameter: wherein , denote the coordinates of two position points, respectively, denotes the Euclidean norm of a vector, i.e. the length of the vector; Since the number of neighbors for each ship is time-varying, it is necessary to aggregate their past trajectories, given a set of neighboring ship trajectories where denotes the th trajectory, containing historical trajectory points from to , denotes the index of the trajectory, from 1 to , denotes the number of time steps to look back, is the total number of trajectories in the set, the aggregated state of the set of neighboring ship trajectories is computed as: Where element-wise addition is applied to preserve the fixed structure of the encoder input and count information, and the concatenated features of the self and neighbors are processed to obtain a more informative representation: Step S203, to create pixel-level alignment between different modalities to learn a rich semantic context representation that effectively utilizes scene context and observed trajectory, first, past trajectories are transformed into trajectory heatmaps using a Gaussian kernel transformed into trajectory heatmaps are computed as follows: wherein, is the standard deviation of the Gaussian kernel, denotes a time stamp the longitude and latitude coordinates at that time stamp, if two Gaussians overlap, element-wise summation is taken to guarantee the intensity and smoothness of the trajectory heat map, a 2D-CNN is used to extract joint features from the scene context and past trajectories, and the trajectory heat map is fused with the image to represent as: wherein is a hyperparameter, is a scene image, denotes an element-wise multiplication operation, and M is a trajectory heat map.

4. The method of claim 1, wherein: The specific steps of the step S30 are as follows: Step S301: The features obtained by the steps S201, S202 and S203 are concatenated and transformed by an MLP layer to obtain a condition aggregator, and the condition aggregator is represented as follows: 。 5. The method of claim 1, wherein: The specific steps of the step S40 are as follows: Step S401, in essence, the inverse diffusion network is a noise prediction network that takes the diffused future trajectories , denoising steps and conditional embedding As input, the diffused trajectories obtained at different time steps are first feature-encoded by computing a temporal embedding representation, denoted as follows: wherein, from fixed samples, with conditions along the variable dimension connected to generate new conditions , input transformed by linear projection to get diffuse future trajectory embeddings ; Step S402, apply several fully connected layers and employ element-wise multiplication to explicitly map the input embedding in the feature dimension and new conditions Up-sampling is performed to obtain an aggregated representation of the features : wherein 、 、 denotes a fully connected layer, is an activation function, by timestamp broadcasting to form , and the same timestamp broadcasting gives ; Step S403: Before the decoder layer, a position embedding is introduced in the implementation to inject information about the relative positions of different timestamps, and then a decoder based on a Transformer is composed of a plurality of self-attention layers to model the complex spatio-temporal dependencies of the fused features, wherein the self-attention module promotes enhanced interaction sequence representation between timestamps by utilizing derived queries, keys and values, and the decoder based on the Transformer is represented as follows: wherein denotes a self-attention module, denotes a position encoding module wherein denotes a feed-forward neural network that down-samples the output sequence to the trajectory dimension by several fully connected layers with the same aggregation operation to form a noise prediction of the same dimension as the denoised output is computed in a deterministic way, denoted as: where is the denoised trajectory representing the estimated trajectory at step , is the noise prediction function that predicts the noise part in the trajectory based on the current diffusion trajectory and the denoising step and the conditional embedding , is the hyperparameter obtained by fixed sampling, is the step size reduction for accelerating the backward diffusion process, is the estimate of the original trajectory state at denoising step , which is calculated as wherein represents the parameter of the diffusion step number k, is the denoised trajectory, representing the trajectory estimate at the denoising step number k.

6. The method of claim 4, wherein: The specific steps of the step S50 are as follows: Step S501: A training loss function is defined as a mean square error loss, and both diffusion and back-diffusion are modeled as Markov chains with Gaussian transitions, and trajectory prediction is performed by modeling a joint distribution, and the modeling is represented as follows: wherein, denotes the joint probability distribution of the initial Gaussian noise trajectory under given conditions to the final trajectory , denotes the probability distribution of the initial Gaussian noise trajectory , denotes a conditional probability distribution, denotes the probability distribution of each intermediate trajectory from to , is the total number of steps in the forward diffusion process, denotes that the initial trajectory is subject to a Gaussian distribution with mean 0 and covariance matrix , is the conditional embedding obtained in step S301, containing the observed trajectories of the target ship itself and neighboring ships, as well as the multi-modal scene context condition information, while: wherein, denotes that this conditional probability follows a normal distribution with mean, variance, denotes the variance obtained by fixing the sampling value, denotes the mean estimate output controlled by the parameters of the inverse diffusion denoising network, denotes the variance obtained by fixing the sampling value, denotes the mean estimate output controlled by the parameters of the inverse diffusion denoising network, denotes the variance obtained by fixing the sampling value. Since the forward diffusion process and the backward diffusion process are both Gaussian distributions, the training target is naturally converted into calculating the difference between the means: wherein denotes following the backward diffusion process, with for a given condition, the approximate posterior mean estimate at denoising step denotes following the backward diffusion process, with for a given condition, the mean estimate given by the backward diffusion denoising network with parameters at denoising step for a given condition, the mean estimate given by the backward diffusion denoising network with parameters at denoising step​​ Finally, the conditional denoiser is learned by minimizing the following loss function: The following simplified loss function is applied to learn the conditional denoiser, which can bring better generation properties as follows: 。 7. The method of claim 1, wherein: The specific steps of the step S60 are as follows: Step S601, the reverse diffusion process adopts a non-Markov denoising strategy DDIM, which starts by reconstructing the original data from the noise state, and then deterministically estimates the direction to the next target timestamp, once the encoder And the decoder is trained, the reasonable trajectory can be generated by the denoising step number, the initial input is Gaussian noise , the iteration number is K / s times, and the denoised trajectory is obtained by iteratively applying the denoising step number with reduced timestamp As the final prediction.

Images