Space-time trajectory prediction method and device based on multi-modal condition potential diffusion generation model

Abstract

The invention discloses a spatio-temporal trajectory prediction method and device based on a multi-modal condition potential diffusion generation model, electronic equipment and a storage medium, and the method comprises the steps: cleaning and aligning multi-source data such as trajectory data, environment semantics and a road network, and constructing standardized input; extracting spatio-temporal context representation through multi-modal attention, mining an environment topological structure in combination with an iterative graph network, and generating structured node embedding; adopting dual-channel cross-modal attention fusion trajectory dynamic features and graph structure information to form unified semantic representation; the conditional variation auto-encoder fuses feature codes to a low-dimensional potential space, conditional reverse denoising generation is executed by using a potential diffusion model, and a future trajectory sequence is recovered step by step; and light weight of the model is realized through diffusion consistency distillation, and reverse sampling is compressed. Through the multimode topology-diffusion distillation integrated architecture, the precision, continuity and reasoning efficiency of trajectory prediction under sparse noise data are improved, and the method is suitable for scenes such as ecological monitoring, intelligent traffic and navigation.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent sensing, spatiotemporal data analysis, multimodal information fusion, and generative prediction technology, specifically relating to a spatiotemporal trajectory prediction method and device based on a multimodal conditional latent diffusion generation model. This method comprehensively utilizes multimodal attention mechanisms, graph structure association modeling, conditional variational autoencoders (CVAEs), and latent diffusion models (LDMs) to uniformly model multi-source heterogeneous sensing data, including trajectory data, environmental semantic information, and spatial topology, and performs conditional spatiotemporal trajectory generation in the latent space. Simultaneously, this invention uses diffusion consistency distillation technology to achieve model lightweighting and inference acceleration, thereby efficiently predicting the future trajectory of a target in complex, noisy, and multi-structure constrained environments. This invention can be applied to ecological monitoring, intelligent transportation, smart city management, unmanned system navigation, and other technical fields requiring high-precision trajectory prediction. Background Technology

[0002] With the widespread adoption of multi-source sensing devices, trajectory data-based behavior analysis and spatial prediction technologies have been widely applied in fields such as ecological monitoring, urban traffic management, public safety assurance, unmanned system navigation, and smart city construction. By analyzing and predicting spatiotemporal data such as animal movement paths and pedestrian or vehicle trajectories, tasks such as habitat assessment, traffic optimization, path planning, and environmental situational awareness can be supported. However, as the complexity of application scenarios continues to increase, trajectory prediction technology still faces many challenges in areas such as reliance on multimodal sensing data, cross-scenario modeling, and real-time inference.

[0003] First, multi-source trajectory data generally suffers from sparsity, discontinuity, and noise interference. In ecological monitoring scenarios, due to limited sensor coverage, highly random target behavior, and power consumption limitations of data acquisition equipment, animal movement trajectories often exhibit breakpoints, jumps, or long periods of missing data. In urban traffic scenarios, due to issues such as positioning accuracy, environmental obstruction, and communication congestion, vehicle or pedestrian trajectories often contain noise points or inconsistent sampling. Traditional trajectory prediction methods based on rules, Kalman filtering, or recurrent neural networks are prone to accumulating errors when faced with incomplete data, making it difficult to form continuous and reliable prediction sequences.

[0004] Secondly, multimodal information fusion still faces significant technical bottlenecks. Trajectory-related environmental semantics, road structures, geographic topology, and sensor data exhibit heterogeneity and cross-domain characteristics, with substantial differences in sampling frequencies and representation formats, increasing the difficulty of unified modeling. Traditional methods for fusing multimodal data typically employ concatenation, weighting, or independent encoding, which struggle to capture the deep interactions between trajectory features and environmental features, resulting in insufficient model adaptability to complex scene changes.

[0005] Furthermore, with the development of deep generative models, methods based on variational inference, generative adversarial networks, and diffusion models are increasingly being applied to trajectory prediction tasks. Although diffusion models demonstrate strong capabilities in handling uncertainty, noise, and missing data, they still have the following limitations:

[0006] (1) High-dimensional diffusion space reasoning is costly and difficult to meet the requirements of real-time prediction;

[0007] (2) The diffusion process requires multiple inversion sampling steps, resulting in a large inference delay;

[0008] (3) The lack of effective modeling of multimodal conditions makes it difficult for the prediction target to strictly follow the environmental semantics and spatial structure constraints.

[0009] Meanwhile, while the Latent Diffusion Model (LDM) can reduce computational cost, its performance is highly dependent on the design of conditional features and the quality of latent variable representation; if the conditional space is not adequately constructed, it is difficult to achieve accurate and stable trajectory generation.

[0010] Furthermore, the consistency distillation technique, proposed in recent years, can compress multi-step diffusion inversion into fewer-step inference, but it still has the following limitations in trajectory prediction tasks:

[0011] (1) Existing consistent distillation is mainly aimed at image or speech generation, and its ability to preserve temporal structure and spatial topology information is insufficient;

[0012] (2) Lack of a multimodal conditional consistency optimization mechanism for trajectory prediction scenarios;

[0013] (3) It is difficult to balance distillation efficiency and trajectory quality, thus limiting its application in real-world scenarios.

[0014] In summary, existing trajectory prediction technologies still struggle to simultaneously meet the following requirements:

[0015] (1) Maintain the continuity and stability of trajectory prediction under sparse, missing and noisy data conditions;

[0016] (2) Simultaneously model trajectory dynamic features, environmental semantic features and spatial structure dependence, and effectively fuse multimodal information;

[0017] (3) Achieve low-latency and high-efficiency reasoning capabilities while maintaining prediction quality;

[0018] (4) Achieve high generalization capability and reliability in complex scenarios.

[0019] Therefore, existing trajectory prediction technologies still have shortcomings in processing multi-source heterogeneous sensing data, addressing trajectory sparsity and missing data, modeling complex spatial structures, and meeting real-time prediction requirements. In particular, there is a lack of effective unified solutions to the problems of high inference overhead in diffusion models, insufficient fusion of multimodal conditions, and difficulty in balancing prediction accuracy and efficiency. There is an urgent need for a new trajectory prediction framework that can stably model uncertainties in the latent space, possess cross-modal structural sensing capabilities, and reduce inference costs while ensuring generation quality, in order to better adapt to the application needs of real-world complex scenarios such as ecological monitoring and intelligent transportation. Summary of the Invention

[0020] I. Purpose of the Invention

[0021] To address the problems of existing trajectory prediction methods, such as sparse multi-source heterogeneous data, severe data gaps, strong noise interference, difficulties in cross-modal information fusion, and insufficient inference efficiency of diffusion models, this invention proposes a spatiotemporal trajectory prediction method and apparatus based on a multimodal conditional latent diffusion generation model. This method constructs a unified multimodal conditional generation framework, realizes conditionally controlled probabilistic trajectory generation in a low-dimensional latent space, and combines it with consistent diffusion distillation to achieve efficient inference, thereby effectively improving the accuracy, robustness, and computational efficiency of trajectory prediction.

[0022] The objectives of this invention include:

[0023] (1) Achieve generative modeling of the uncertainty of future trajectories and overcome the error accumulation problem of traditional deterministic prediction methods.

[0024] (2) Realize cross-modal structural perception information fusion, so that trajectory data and heterogeneous information such as environmental context and spatial topology can be efficiently coupled in a unified semantic space.

[0025] (3) While effectively reducing inference overhead, it maintains prediction accuracy and meets the real-time and robust requirements of scenarios such as ecological monitoring and intelligent transportation.

[0026] II. Technical Solution

[0027] This invention proposes a spatiotemporal trajectory prediction method and apparatus based on a multimodal conditional latent diffusion generation model, aiming to address the problems of insufficient accuracy and stability in trajectory prediction under conditions such as sparse multi-source data, significant noise interference, complex spatiotemporal structure, and insufficient real-time inference efficiency in existing technologies. This invention constructs a unified multimodal conditional probabilistic generation framework, combining multimodal attention representation learning, graph structure association modeling, cross-modal interactive fusion, joint modeling of conditional variational autoencoder (CVAE) and latent diffusion model (LDM), and a diffusion consistency distillation strategy to achieve high-precision, high-robustness, and high-efficiency prediction of future target trajectories in complex scenarios.

[0028] (I) Trajectory Prediction Methods

[0029] The trajectory prediction method based on a multimodal conditional latent diffusion generation model provided by this invention includes three main stages: training stage, model distillation stage, and inference stage; the device embodiment uses a distilled student model, which only contains inference-related modules.

[0030] (1) Model training phase

[0031] The training phase is used to build the basic generative model, and its forward computation is consistent with the inference phase, including steps S1 to S7. After each forward inference, a loss function is constructed based on the output trajectory and training samples, and parameter updates are performed.

[0032] Step S1: Multi-source heterogeneous data acquisition and preprocessing

[0033] Multi-source heterogeneous sensing data collected under various environmental backgrounds is acquired, including core trajectory data, environmental semantic context data, weather information, and road network structure data. Trajectory data consists of a series of timestamped geospatial coordinates; environmental context data includes multimodal sensing information such as images, semantic labels, acceleration, attitude, and geographic attributes. The above data undergoes preprocessing, noise filtering, normalization, time synchronization, and coordinate system alignment to construct a unified input feature space.

[0034] Step S2: Spatiotemporal context representation learning for multimodal attention

[0035] This study utilizes a multimodal attention mechanism to jointly extract spatiotemporal features from heterogeneous input data. Through operations such as trajectory aggregation, adaptive trajectory alignment, and multi-head self-attention encoding, it captures the temporal dynamic features of the trajectory itself and the multimodal dependencies between the trajectory and the environment, generating a compact and highly representative spatiotemporal context vector.

[0036] Step S3: Structural Association Modeling Based on Iterative Graph Networks

[0037] An unlabeled node graph is constructed, and the adjacency matrix is ​​initialized using metrics such as cosine similarity and mutual information. Furthermore, the adjacency matrix is ​​iteratively optimized using a Graph Attention Network (GAT) combined with a contrastive learning strategy. This robustly mines the complex topological structures and relationship patterns hidden in cross-modal data, generating high-quality, robust structured node embeddings to effectively enhance the sensitivity of trajectory prediction to complex spatial structures.

[0038] Step S4: Dual-channel cross-modal attention fusion

[0039] A dual-channel cross-modal attention structure is adopted to deeply fuse high-dimensional trajectory features with environmental context features. A cross-attention layer enables bidirectional complementary enhancement and information interaction between features, while a self-attention layer maintains the temporal or structural consistency within each modality, ultimately resulting in a complete, interpretable, and robust cross-modal spatiotemporal interactive representation.

[0040] Step S5: Conditional probability generation based on CVAE and LDM

[0041] In this step, a conditional variational autoencoder is first used to encode the cross-modal spatiotemporal features in a low-dimensional latent space, obtaining the distribution of conditional latent variables used for trajectory generation. Then, a latent diffusion model is applied to this latent variable space, progressively injecting noise through forward diffusion and performing conditional backsampling using an inverse denoising network to recover the conditional distribution of the latent variables and generate the corresponding trajectory point sequence. This step also introduces a binary masking mechanism to identify missing points or incomplete segments in the input trajectory, enabling the diffusion generation process to maintain stable reconstruction even under missing data conditions and improving the model's prediction robustness in sparse trajectory scenarios.

[0042] Step S6: Trajectory Reconstruction Based on Predicted Output

[0043] During the training phase, based on the trajectory output generated in step S5, and combined with road network structure, motion feasibility constraints, and environmental semantic information, necessary post-processing and rationality verification are performed on the predicted trajectory to enhance structural consistency during training. This prediction result is only used for loss calculation during the training phase and is not used as the actual output in the final deployment phase.

[0044] Step S7: Loss Function Construction and Parameter Update

[0045] During training, this invention constructs a spatiotemporal awareness loss function based on the forward inference results of steps S1–S6, used for joint optimization of the model. This loss includes a spatial structure constraint term based on the spatial co-occurrence matrix and a temporal dynamic constraint term based on cross-entropy or sequence consistency metrics. An adaptive weight adjustment mechanism achieves a dynamic balance between spatial and temporal errors to adapt to the characteristics of trajectory data under different noise levels and sampling densities. After loss calculation, gradient backpropagation is used to jointly update the parameters of the multimodal attention network, graph structure embedding network, conditional variational autoencoder, and latent diffusion generation network end-to-end until the overall model converges. The trained model serves as the teacher model for subsequent distillation stages.

[0046] (2) Model Distillation Phase

[0047] After the teacher model is trained and its parameters are fixed, this invention further employs a Diffusion Consistency Distillation (DCD) strategy to compress and accelerate the model, thereby improving inference efficiency. In the distillation stage, the consistency loss between the teacher and student models in the noise estimation space and the potential diffusion dynamics is constructed. The parameters of the student model are then updated, effectively reducing the inverse denoising sampling steps while maintaining the quality of trajectory generation. This compresses the original high-step diffusion inversion process to at least a few sampling steps. This stage only optimizes the student model and does not update any parameters of the teacher model, resulting in a distilled model that is more advantageous in terms of computational resource consumption and inference speed for practical deployment.

[0048] (3) Reasoning stage

[0049] In the inference phase, this invention performs trajectory generation based on the distilled student model, performing only forward inference without gradient calculation. The inference process corresponds to the forward part of steps S1 to S5, including multi-source data input, multimodal attention representation learning, graph structure association modeling, cross-modal feature fusion, and potential diffusion condition generation. The generated trajectory is further post-processed by combining road network topology, motion feasibility constraints, and environmental context information, ultimately outputting a continuous, structurally consistent, and temporally logical future spatiotemporal trajectory, which can be widely applied to various trajectory prediction scenarios such as ecological monitoring, intelligent traffic scheduling, robot autonomous navigation, and UAV trajectory planning.

[0050] (II) Trajectory Prediction Device

[0051] Based on the same technical concept as the aforementioned methods, this invention provides a trajectory prediction device for performing multi-source data fusion, latent spatial condition generation, and future trajectory inference. This device uses a student model obtained through training and distillation, containing only the modules required for inference execution, and does not include any training or distillation operations. The device may include:

[0052] (1) Multi-source heterogeneous data acquisition and preprocessing module

[0053] This module is used to collect and process multi-source heterogeneous sensing data, including target trajectory data, environmental semantic context, weather conditions, and road network structure information. The module performs denoising, normalization, time synchronization, and spatial coordinate alignment on the raw data to construct a unified and standardized input feature space, ensuring the validity and consistency of the model input.

[0054] (2) Context Representation Learning Module

[0055] This module extracts spatiotemporal features from preprocessed input data based on a multimodal attention mechanism, including steps such as trajectory aggregation, adaptive trajectory alignment, and multi-head self-attention encoding, to capture the temporal dynamic patterns of trajectories and their relationship with the environmental context, forming a compact and semantically consistent spatiotemporal context representation.

[0056] (3) Graph Structure Association Modeling Module

[0057] This module employs an iterative graph network (such as the Gaussian Attention Network, GAT) to model the structured relationships of the input multimodal data. By constructing a node graph, updating the adjacency matrix, and performing graph embedding, it extracts the latent spatial topological relationships between trajectory points and between them and the environmental structure, forming a node embedding representation that reflects spatial structure dependencies. This module enhances the model's sensitivity to complex spatial layouts and environmental structures.

[0058] (4) Cross-modal attention fusion module

[0059] This module, based on a dual-channel cross-modal attention structure, deeply fuses the spatiotemporal context features from step S2 and the graph structure embedding from step S3. It achieves bidirectional information interaction between trajectory features and environmental features through a cross-attention mechanism, and maintains consistency within each modality through a self-attention mechanism, thereby obtaining a complete, robust, and interpretable cross-modal spatiotemporal interactive representation. This module is a key link connecting high-dimensional multimodal features with latent conditional generative models.

[0060] (5) Conditional Probability Generation Model Module

[0061] This module first uses a Conditional Variational Autoencoder (CVAE) to map the cross-modal fused spatiotemporal features to a low-dimensional latent space, generating conditional latent variables to constrain the generation process. Then, it directly performs inverse conditional denoising sampling within the Latent Diffusion Model (LDM), progressively inverting the initial noise latent variables to obtain the latent representation corresponding to the future trajectory point sequence. The entire generation process only includes the backsampling step in the inference phase, eliminating the forward noise addition operation in the training phase. Simultaneously, this module incorporates a binary masking mechanism to identify and compensate for missing trajectory segments or low-confidence points in the input data, ensuring the generation process maintains prediction stability and robustness even with incomplete input.

[0062] (6) Trajectory Output Module

[0063] This module performs constraint verification and smoothing on the trajectory output by the conditional generation model. By combining road network topology, motion feasibility constraints, and environmental semantic information, it generates continuous, structurally consistent, and temporally logical future trajectories to meet the application needs of scenarios such as ecological monitoring, intelligent traffic management, autonomous driving, and robot navigation.

[0064] (iii) Electronic devices, computer-readable storage media and computer program products

[0065] Based on the same technical concept as the methods and apparatus described above, the present invention also provides various computing device configurations that can be used to carry out or execute the methods of the present invention. In one exemplary embodiment, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the electronic device performs all the steps of the trajectory prediction method defined in the first aspect of the present invention, or any combination thereof.

[0066] In another embodiment, the present invention provides a non-transitory computer-readable storage medium storing computer program instructions. When these instructions are read and executed by a processor, the computer performs the trajectory prediction process described in the first aspect of the present invention, thereby enabling the portability and deployment of the method functionality without additional hardware modifications.

[0067] In a further embodiment, the present invention provides a computer program product comprising computer-executable instructions. When the instructions are executed in a computer or processor device, they perform the steps described in the present invention, including multi-source data processing, spatiotemporal representation learning, cross-modal relationship modeling, latent space generation, and trajectory output, thereby achieving generative prediction of future trajectories.

[0068] In summary, this invention constructs a unified spatiotemporal prediction framework based on a probabilistic generative model. It deeply couples and collaboratively optimizes key technologies such as multimodal attention mechanisms, graph structure association modeling, cross-modal attention fusion, conditional latent variable generation models, and diffusion-consistent distillation, fundamentally overcoming the limitations of traditional trajectory prediction methods in terms of data gaps, noise interference, and inference efficiency. Through global joint modeling and latent space generation mechanisms, this invention ensures effective improvements in the accuracy, continuity, and robustness of trajectory prediction results, providing reliable and efficient technical support for applications such as ecological monitoring, intelligent transportation, unmanned system navigation, and urban management.

[0069] III. The Innovative Nature of the Invention

[0070] This invention addresses the core challenges of sparse, incomplete, and noisy multi-source heterogeneous trajectory data, as well as the difficulty of existing methods in simultaneously achieving prediction accuracy, robustness, and real-time inference efficiency. It proposes a unified trajectory prediction framework based on a multimodal conditional latent diffusion generation model. Through innovations in mechanism, structure, and strategy, this invention achieves high-fidelity and robust prediction of complex moving target trajectories. Its innovation is mainly reflected in the following three aspects:

[0071] (1) A unified multimodal conditional latent diffusion generation framework enables high-precision modeling of trajectory uncertainties.

[0072] This invention is the first to construct a unified, end-to-end trainable generative trajectory prediction framework that deeply integrates a multimodal feature encoder, a cross-modal structure perception module, and a conditional latent diffusion model (CVAE-LDM). The core advantage of this framework lies in using CVAE to compress high-dimensional spatiotemporal features into a low-dimensional latent space, and using this latent encoding as a condition to drive the LDM inverse denoising and generation process. By performing probabilistic inference in the low-dimensional latent space, the model can intrinsically simulate the randomness, multimodality, and environmental uncertainty of target motion. This fundamentally differs from traditional deterministic models, effectively improving the continuity, diversity, and overall robustness of prediction results, while simultaneously avoiding the cascading amplification of errors in traditional staged methods through global optimization.

[0073] (2) Based on the conditional guidance mechanism of cross-modal structure awareness, the interpretability and robustness of the generative model are effectively improved.

[0074] This invention innovatively designs a cross-modal structure-aware conditional guidance mechanism, focusing on constructing a high-quality conditional space that can directly drive the latent diffusion model. This mechanism is achieved through the synergistic effect of a multimodal attention network and an iterative graph structure modeling: the attention network is responsible for capturing the temporal dependencies between trajectory dynamics and environmental semantics; the iterative graph network (IGNN) is responsible for learning and embedding spatial topology and neighborhood constraints; subsequently, a dual-channel cross-modal attention structure achieves bidirectional deep fusion and enhancement of trajectory features and environmental features. This multi-level fusion ensures that the generated conditional representation not only understands motion patterns and semantic context but also possesses sensitivity to spatial structure, providing sufficient, accurate, and strongly constrained guidance information for the subsequent CVAE-LDM latent generation process.

[0075] (3) A lightweight consistency diffusion distillation strategy for real-time applications to achieve efficient inference.

[0076] To address the challenges of data sparsity, incompleteness, and high real-time requirements in real-world scenarios, this invention integrates a series of efficient inference and robustness strategies. First, a Diffusion Consistency Distillation (DCD) strategy is introduced to reduce the backsampling steps of the latent diffusion model, thereby achieving model lightweighting and inference acceleration to meet the demands of high-frequency, real-time inference. Second, a binary masking mechanism enhances the model's adaptive handling of missing trajectory data and low-confidence points. Finally, a spatiotemporal-aware joint loss function is designed, dynamically balancing and optimizing structural constraints based on the spatial co-occurrence matrix and temporal consistency constraints based on cross-entropy, effectively improving the model's generalization ability and prediction accuracy under different noise levels, sampling densities, and complex environments.

[0077] IV. Technical Effects

[0078] Compared with existing trajectory prediction technologies, this invention constructs a unified multimodal conditional latent diffusion generation framework and combines a cross-modal structure perception mechanism with a diffusion consistency distillation strategy, thereby effectively improving the trajectory prediction technology in terms of accuracy, robustness, generalization ability, and inference efficiency. The specific improvements are as follows:

[0079] (1) Effectively improves the accuracy, continuity and stability of trajectory prediction

[0080] This invention obtains a more complete and semantically consistent spatiotemporal conditional representation through multimodal attention, graph structure association modeling, and cross-modal interaction mechanisms. It then performs conditional diffusion denoising generation in a low-dimensional latent space, effectively enhancing the model's robustness to noise disturbances, uneven sampling, and missing trajectories. In complex environments (such as severe occlusion, sparse samples, or the presence of outliers), this invention can generate future trajectories that are temporally logically consistent, spatially smooth, and closely approximate the actual motion trend, effectively reducing common prediction distortion problems such as jumps, drifts, and breaks.

[0081] (2) Enhance the model’s generalization ability and cross-modal perception ability in complex scenarios.

[0082] By incorporating trajectory dynamics, environmental semantic information, spatial topology, and scene context into the conditional generation space, this invention achieves enhanced cross-modal structure perception capabilities, enabling the model to understand implicit structural consistency across different environments. Even under conditions of complex road layouts, drastic scene changes, or incomplete environmental elements, the model can still stably identify target motion patterns and maintain high-quality predictions, effectively improving generalization performance in cross-regional, cross-scene, and even cross-modal applications.

[0083] (3) Achieve efficient real-time inference while maintaining prediction quality

[0084] This invention effectively compresses the backsampling steps of the latent diffusion model through a diffusion consistency distillation strategy, significantly reducing inference costs. The distilled student model retains the expressive power of the original diffusion generation framework but requires only a few sampling steps to generate a complete future trajectory, effectively improving inference speed compared to the basic diffusion model. This characteristic enables this invention to meet the stringent requirements of "real-time performance" and "high throughput" in scenarios such as unmanned systems, mobile terminals, and high-frequency traffic prediction, maintaining reliable prediction performance even on resource-constrained devices. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings referenced in the embodiments of the specification are briefly described below. It should be understood that these drawings are only used to illustrate some embodiments of the present invention and should not be considered as limiting the scope of protection of the present invention. Those skilled in the art can obtain other forms of drawings based on these drawings without creative effort.

[0086] Figure 1 This is a schematic diagram of the overall process of the trajectory prediction method based on a multimodal conditional latent diffusion generation model provided in an embodiment of the present invention.

[0087] Figure 2 This is a schematic diagram of the trajectory prediction device based on a multimodal conditional latent diffusion generation model provided in an embodiment of the present invention.

[0088] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and beneficial effects of this invention clearer, exemplary embodiments of the invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention. Without departing from the spirit of this invention, those skilled in the art can make various modifications and substitutions to the embodiments according to specific application scenarios.

[0090] It should be noted that the data involved in the embodiments of the present invention (including but not limited to trajectory data, environmental semantic information, structured map data, etc.) should be obtained and used in compliance with relevant laws and regulations.

[0091] This chapter, in conjunction with the accompanying drawings, provides a detailed description of exemplary embodiments of the present invention. The following description strictly follows the three-stage structure of the present invention: "training stage → distillation stage → inference stage," and integrates the method modules (multimodal attention, graph structure embedding, cross-modal fusion, CVAE, LDM, repaint resampling, and spatiotemporal loss) into the process in a logical order to avoid repetition and contradictions.

[0092] I. Overview of the Overall Method and Process

[0093] like Figure 1 As shown, the present invention proposes a trajectory prediction method based on a multimodal conditional latent diffusion generation model. It takes multi-source heterogeneous sensing data as input and sequentially goes through multimodal representation learning, graph structure association modeling, cross-modal feature fusion, conditional probability generation, training optimization and diffusion consistency distillation processing. Finally, it generates a continuous, feasible future trajectory that conforms to environmental semantic constraints in the inference stage.

[0094] The overall approach is divided into three interrelated stages according to the functional flow, with each stage having a clear role and well-defined boundaries:

[0095] (1) Training phase (S1–S7)

[0096] The training phase aims to learn a unified spatiotemporal feature representation from multi-source trajectory and environmental data, and to establish a stable and controllable conditional generative model in the latent space. This phase includes steps such as data preprocessing, multimodal attention representation learning, graph structure modeling, cross-modal fusion, CVAE latent variable encoding, LDM diffusion generation, and spatiotemporal perception loss optimization, to achieve joint modeling of trajectory dynamics and environmental structure and end-to-end parameter updates.

[0097] (2) Distillation stage

[0098] To improve inference efficiency, this invention employs the Diffusion Consistency Distillation (DCD) method, compressing the multi-step back diffusion process in the teacher model into a student model requiring only a few inference steps. This stage does not change the teacher model parameters but optimizes the student model separately, effectively reducing computational overhead while maintaining generation quality, thus achieving fast and lightweight trajectory generation capabilities.

[0099] (3) Reasoning stage

[0100] The inference phase only executes the forward steps from the training phase, including data preprocessing, multimodal attention representation learning, graph structure modeling, cross-modal fusion, and inverse denoising generation of the latent diffusion model (S1–S5). This phase no longer includes noise addition, loss function calculation, parameter updates, or trajectory reconstruction optimization. Instead, it directly uses a lightweight student model to predict future trajectories and finally performs necessary constraint checks by combining road topology and environmental semantics.

[0101] Based on the above three-stage structure, the functions, processes and implementation methods of steps S1–S7 will be described in detail in subsequent chapters so that those skilled in the art can clearly understand the overall technical logic and implementation mechanism of the present invention.

[0102] II. Training Phase

[0103] Steps S1–S7 of this invention constitute the core computational flow of the training phase. During the training phase, multimodal representation learning, graph structure embedding, cross-modal fusion, latent variable encoding, diffusion generation, and spatiotemporal awareness loss are jointly optimized end-to-end to obtain a conditional diffusion generation model capable of stable operation in complex environments. Building upon this, this invention further introduces a diffusion consistency distillation mechanism to effectively improve backsampling efficiency during the inference phase. The following is combined with… Figure 1 The main steps of the training phase are explained.

[0104] S110: Data Acquisition and Preprocessing

[0105] This step acquires multi-source heterogeneous sensing data from the real environment, including the target's historical trajectory, environmental semantic information, road network structure, weather and geospatial attributes, etc. Simultaneously, the time period to be predicted is determined based on the prediction task. All input data undergoes preprocessing operations such as formatting, time synchronization, coordinate alignment, noise suppression, and interpolation completion to construct a unified input feature space, providing a reliable foundation for subsequent model processing.

[0106] S120: Spatiotemporal Context Representation Learning Based on Multimodal Attention

[0107] This step aims to construct a high-quality spatiotemporal representation for conditional generation. The main process is as follows:

[0108] (1) Trajectory clustering and representative trajectory aggregation

[0109] The DTW distance is used to calculate the similarity between trajectories, and the K-centroid clustering algorithm is used to cluster similar trajectories, mining representative historical animal trajectories as candidates for future paths. Specifically, given two trajectories X = {x1, x2, ..., x...} n} and Y = {y1, y2, ..., y n}, DTW distance is defined as:

[0110]

[0111] Here, π represents the aligned time point sequence found through dynamic programming that minimizes the total distance. However, DTW trajectory clustering needs to consider the length variations and time offsets between different trajectories, resulting in high computational complexity. To address this issue, this project introduces the K-medoids clustering algorithm, grouping trajectories into k clusters with similar movement patterns, and selecting representative real trajectories from each cluster as the initial trajectories for the trajectory group. This approach reduces data redundancy and more robustly captures common patterns in group movement. Furthermore, when the movement behavior of an individual to be predicted differs significantly from existing trajectory patterns, the system treats that individual's current trajectory as an independent trajectory. Given an individual trajectory e, its prefix trajectory e c Using the keyword as a key, the most similar trajectory e is found by calculating its DTW similarity with all trajectories in the trajectory group. sim If DTW(e1,e2) is greater than a predefined threshold θ, it indicates a significant difference between the individual trajectory and the existing pattern, requiring its addition to a new trajectory group to further optimize the model's generalization ability in subsequent target movement predictions. Subsequently, to effectively capture and represent the key features of individual movement, a multi-head self-attention mechanism of a Transformer network is used to capture individual movement at different time steps. This trajectory encoder projects the movement features into a more compact and higher-dimensional embedding dimension, obtaining the trajectory spatiotemporal representation e. tra .

[0112] S130: Iterative Graph Network Embedding

[0113] This step is used to automatically mine potential topological relationships from multimodal environmental data and generate a stable and robust graph structure representation. The system first constructs an initial unlabeled node graph based on similarity metrics and mutual information. Then, it continuously updates the relationships between nodes using an iterative optimization strategy of the adjacency matrix and a graph attention network, ensuring its reliability even under conditions of noise interference, missing data, and structural complexity.

[0114] When constructing the initial unlabeled node graph, the system automatically classifies positive and negative neighbors by calculating the similarity between nodes (such as cosine similarity, Jaccard coefficient, etc.) and mutual information, and determines potential connecting edges based on statistical thresholds or classifiers. Subsequently, by iteratively optimizing the similarity function, the boundary partitioning gradually converges to a stable state, thereby obtaining the preliminary adjacency matrix.

[0115] During the adjacency matrix iteration phase, the system inputs environmental features into the graph attention network and estimates the strength of connections between nodes through attention weights. Based on the attention score and a threshold, potential connecting edges are classified into positive, negative, and unlabeled edges. A contrastive learning mechanism is then used to maximize the discriminative power between positive and negative samples, allowing the adjacency matrix to continuously approach its optimum during iteration. Iteration continues until all potential edges are explicitly labeled, ultimately forming a complete and optimized adjacency matrix.

[0116] Through the aforementioned adaptive graph construction and iterative optimization process, this step learns discriminative structural associations from high-dimensional, noisy, and partially missing data, forming a robust and stable graph embedding representation. This provides structurally consistent environmental context features for subsequent cross-modal fusion and trajectory generation. Finally, the system generates low-dimensional embedding vectors for nodes based on the optimized adjacency matrix, achieving a unified representation of multimodal structural information.

[0117] S140: Cross-modal attention fusion

[0118] The cross-modal attention module aims to combine individual trajectories with multimodal environmental information to enhance the richness and representativeness of feature representations by exploring the interrelationship between individual movement and environmental context. The module employs a dual-channel structure: one channel is used to mine features where individual movement is influenced by environmental context, and the other channel is used to extract environmental context information related to individual movement.

[0119] The core of the cross-modal attention module lies in the design of the cross-attention layer and the self-attention layer. The cross-attention layer is responsible for interactively modeling individual trajectories and environmental information, treating the individual trajectory and environmental information as query sequences and key-value sequences respectively, and adjusting their representations based on their interdependencies. Unlike traditional cross-attention, the dual-channel structure introduces bidirectional relationship modeling, ensuring that the influence of the key-value sequence on the query sequence is not ignored when capturing the interaction information between the individual trajectory and the environmental context, thus obtaining more comprehensive interactive features.

[0120] Subsequently, the environmental context and individual trajectories output by the cross-attention layer are processed by a self-attention layer to calculate the internal correlation of each element within its own sequence, helping the model better understand the contextual information of each input element in its original sequence. Through repeated iterations, this module ultimately generates a cross-modal representation. in Characteristics representing an individual's trajectory Features that represent the context of the environment.

[0121] When calculating the deviation between an individual trajectory and the true trajectory, the individual trajectory feature representation is used. The last element, combined with environmental context features. The final feature representation is obtained through the output of the global pooling layer. The specific formula is as follows:

[0122]

[0123] in This indicates a vector connection, and MLP (Multi-Layer Perceptron) represents a multi-layer perceptron.

[0124] S150: Conditional Probability Generation Based on CVAE and LDM

[0125] (1) Spatiotemporal trajectory latent space encoding based on CVAE

[0126] CVAE (Latent Spatial Encoding) is used to perform latent spatial encoding on input spatiotemporal trajectory data, effectively capturing the spatiotemporal characteristics of the trajectory data and compressing it into a low-dimensional feature representation. This method improves the model's efficiency in processing spatiotemporal data by handling noise in the trajectory data. CVAE's latent spatial encoding also includes discretization of departure time information, enhancing the model's sensitivity to time variations.

[0127] To enhance information capture capabilities during the encoding process, this invention introduces a spatiotemporal context representation based on a multimodal attention mechanism as the conditional input for CVAE. This process is optimized using the following loss function:

[0128]

[0129] Where X is the spatiotemporal trajectory data sequence, C is the spatiotemporal context representation of the multimodal attention mechanism, and z is the latent variable. This loss function minimizes the reconstruction error and the KL divergence of the latent space, enhancing the model's ability to capture and reconstruct trajectory data while ensuring the diversity of latent representations.

[0130] To address incomplete trajectory data, a binary mask sequence M is introduced to indicate the presence of location information at a given time point, thus handling the data gap issue. By introducing the binary mask, this method can effectively handle missing data at certain time points, thereby improving the model's robustness and prediction accuracy.

[0131] (2) LDM diffusion generation

[0132] After completing the latent spatial encoding of CVAE, a latent diffusion model is used for spatiotemporal trajectory prediction. The LDM generates the prediction result by progressively diffusing Gaussian noise into the trajectory data and then reconstructing the trajectory through the reverse process.

[0133] The diffusion process is described as follows:

[0134]

[0135] in It is a potential trajectory variable in the diffusion step, β k The degree of noise addition is controlled. The reverse denoising process is based on conditional input, calculating the conditional probability distribution. The noise is gradually removed to restore the trajectory prediction results.

[0136] To improve the spatial continuity and temporal dependence of predictions, this invention introduces enhanced observations. Interpolation is used to obtain data, thus providing stronger spatiotemporal consistency. By evaluating different interpolation techniques (such as linear interpolation, spline interpolation, etc.), the interpolation method best suited to the data characteristics and prediction task can be selected.

[0137] The denoising network is improved by using a noise estimation model based on a multimodal attention mechanism to capture the spatiotemporal correlation in trajectory data, thereby further optimizing the utilization efficiency of conditional information and the prediction accuracy.

[0138] The mathematical expression for the reverse process is:

[0139]

[0140] Where μ θ and ∑ θ This is the parameterized output of the noise estimation model.

[0141] (3) Resampling techniques based on repaint strategy

[0142] This invention proposes a scheme to optimize the quality of spatiotemporal trajectory prediction data. The method models trajectory prediction as a special form of image filling task. By introducing inpainting technology, in each step of the inverse process of the diffusion model, random sampling is used to sample known trajectories and missing unknown trajectory data. The specific sampling formula is as follows:

[0143]

[0144] Where M tThis is a mask used to combine known and unknown data, ensuring dynamic adjustment and generation of new data in each iteration. By feeding the output of the current step back into the forward process and performing inverse prediction again, the model can achieve iterative optimization of trajectory data by increasing randomness.

[0145] The core advantage of this resampling technique lies in its ability to effectively capture and recover the inherent structure and dependencies of spatiotemporal data, adapting to complex missing patterns. By dynamically adjusting known and unknown trajectory data, it ensures that the generated prediction results are highly consistent with existing data and possess natural fluency, thereby improving prediction performance.

[0146] S160: Trajectory Reconstruction Based on Predicted Output

[0147] During the training phase, this step verifies the structural consistency and physical plausibility of the trajectory results generated by the potential diffusion model to enhance the overall trainability and stability of the model. The system performs a series of post-processing and reconstruction operations on the generated trajectories, including but not limited to the following:

[0148] First, a feasibility constraint check is performed on the predicted trajectory to assess whether the prediction results meet basic motion constraints in terms of physical kinematics, velocity variation range, and trajectory continuity, avoiding non-physical jumps or discontinuities. Second, based on the road network structure or spatial topology of the input scene, the generated trajectory points are aligned with the road network to ensure that the prediction results conform to the structural constraints of the real environment, guaranteeing that the trajectory maintains spatial traversability and location legitimacy. Furthermore, to further improve the continuity and dynamic consistency of the trajectory, this step also performs smoothing and motion trend correction on the predicted trajectory, eliminating minor noise and enhancing the natural changes in the trajectory over time.

[0149] It is important to emphasize that this step is only used to calculate the loss function during the training phase, aiming to improve the model's ability to learn spatial constraints and motion logic through structured feedback signals. The reconstruction results described above will not be directly used as the final output of the inference phase, nor will they affect the forward generation process during the inference phase.

[0150] S170: Spatiotemporal Awareness Loss and End-to-End Optimization

[0151] Traditional loss functions often impose excessive penalties on geographically adjacent outputs, leading to discontinuities in the model's spatial representation. To address this issue, this invention designs a spatiotemporally aware loss function that fully considers the geographical structure of cities or regions during the optimization process, reducing penalties for geographically adjacent locations and enhancing the model's spatial continuity.

[0152] This spatiotemporal perception loss function combines cross-entropy loss and spatiotemporal coefficient adjustment mechanism, and introduces a spatiotemporal relationship balancing hyperparameter λ.s and λ t Achieve space-time awareness optimization. The loss function formula is as follows:

[0153] L ST =λ s ·L spatial +λ t ·L temporal

[0154] Where L spatial This is the spatial loss calculated based on the geographic location co-occurrence matrix, used to maximize the model's prediction accuracy for real geographic location IDs. This part of the loss enhances the model's ability to identify neighboring geographic locations by calculating the spatial-temporal relationship between the model output and the real location IDs. temporal It is the standard cross-entropy loss, used to measure the difference between the output result and the true location ID.

[0155] By designing this space-time-aware loss function, this invention effectively improves the continuity and accuracy of the model in space and time. Especially when dealing with complex urban areas or multimodal data, it effectively reduces the model's error penalty for the output results and enhances the representation ability of spatiotemporal sequences.

[0156] III. Diffusion-Consistent Distillation Stage

[0157] After completing the training phase described above, this invention further employs a Diffusion Consistency Distillation (DCD) mechanism to accelerate inference of the latent diffusion model. This phase uses the trained teacher model as a fixed reference and constructs consistency constraints between the teacher and student models during the multi-step to few-step backsampling process, enabling the student model to maintain generation quality comparable to the teacher model while effectively reducing sampling steps.

[0158] During the distillation process, the teacher model parameters remain frozen and are not updated; the system only optimizes the student model parameters. The teacher model generates an intermediate state sequence through multi-step back diffusion, while the student model directly predicts the corresponding state within a small number of steps. By minimizing the consistency error between the two models in the latent space, the student model can learn the convergence trajectory and denoising rules of the teacher model, thereby achieving the following during the inference phase:

[0159] Fast reverse sampling: The original inversion process of dozens of steps can be compressed into a minimum of a few steps;

[0160] High consistency output: The latent trajectories generated by the student model are highly consistent with those of the teacher model;

[0161] Extremely low inference latency: effectively reduces computing power consumption, enabling models to be deployed and run on edge devices or in real-time scenarios.

[0162] By employing diffusion-consistent distillation, this invention achieves lightweight and efficient prediction models for potential diffusion trajectories without sacrificing generation quality and prediction stability, laying the foundation for subsequent real-time inference stages.

[0163] IV. Reasoning Stage

[0164] The inference phase of this invention aims to achieve fast, low-latency trajectory prediction using a lightweight student model that has been trained and optimized by diffusion-consistent distillation. In this phase, the system only executes the forward pass corresponding to the training phase, omitting noise injection, gradient updates, loss calculations, and training-related trajectory reconstruction steps. The inference phase directly outputs future trajectories that satisfy structural and semantic constraints through forward transformation of multimodal inputs and inverse denoising generation of the latent diffusion model.

[0165] S110: Data Preprocessing

[0166] The inference phase begins with standardization of the input data, including coordinate alignment of trajectory prefixes, time synchronization, format standardization, and necessary noise suppression. This process ensures that the structure of the input data remains consistent with that of the training phase, enabling the model to stably receive trajectory prefixes, semantic maps, road topology, and other multimodal information.

[0167] S120: Multimodal Attention Representation Learning

[0168] The preprocessed input data is fed into a trained multimodal attention encoder, which models trajectory temporal dependence, semantic context, and local change patterns through a self-attention mechanism. The model automatically focuses on the most critical trajectory segments and environmental features for the current prediction, generating spatiotemporal representations for subsequent conditional generation.

[0169] S130: Graph Structure Modeling

[0170] During the inference phase, the system directly calls the stable graph embedding model obtained during the training phase. Based on the environmental topology, spatial adjacency relationships, and semantic structure, it infers the graph structure vector related to the current input. Instead of updating the adjacency matrix at the inference end, the system uses a fixed graph representation, thus ensuring structural consistency and computational efficiency.

[0171] S140: Cross-modal fusion

[0172] The trajectory representation and graph structure representation are then fused in the cross-modal attention module. The model generates a unified cross-modal conditional vector based on the relationship between trajectory dynamics and environmental conditions, enabling the subsequent diffusion generation process to simultaneously utilize multi-source information such as individual motion features, environmental semantics, and spatial structure. This fusion improves the sensitivity of the conditional diffusion model to environmental constraints.

[0173] S150: Conditional LDM Reverse Generation

[0174] During the inference phase, the latent diffusion model only performs the inverse denoising process and does not include a forward denoising step. Starting from randomly initialized latent noise, guided by cross-modal conditional vectors, the system performs a small number of inverse inference steps through a distilled lightweight student model to gradually recover the latent representation of future trajectories and decode them into a continuous sequence of trajectory points. Because distillation reduces the number of sampling steps, the inference process has extremely low latency while maintaining high-quality generation capabilities.

[0175] S160: Trajectory Constraints and Final Trajectory Generation

[0176] After potential trajectories are generated, the system performs lightweight rationality checks and consistency adjustments. This includes:

[0177] (1) Check the legality of road topology to ensure that the predicted trajectory is located in a passable area or conforms to the road network structure;

[0178] (2) Kinematic constraint check to avoid sudden changes in velocity and acceleration that do not conform to the laws of physics;

[0179] (3) Environmental semantic filtering to prevent the trajectory from entering restricted areas, obstacle areas or risk areas;

[0180] (4) Trajectory smoothing: Lightweight filtering method is used to improve trajectory continuity and stability;

[0181] (5) Final trajectory generation: The route that satisfies continuity, feasibility and semantic consistency is output as the final prediction of the reasoning stage.

[0182] Through the above steps, this invention achieves efficient trajectory prediction based on a multimodal conditional latent diffusion generation model in the inference stage. It has advantages such as strong real-time performance, low time consumption, high robustness, and strong deployability, and can be applied to a variety of complex application scenarios such as ecological monitoring, intelligent transportation, smart city management, and unmanned system navigation.

[0183] V. Implementation Method of the Device

[0184] A second aspect of the present invention provides an apparatus for executing the trajectory prediction method based on a multimodal conditional latent diffusion generation model described above. This apparatus can be implemented by software, hardware, or a combination of both, and includes multiple functional modules capable of communicating with and cooperating with each other. It should be understood that the following module division is based solely on functional description, and in specific implementations, these modules can be merged, split, or reorganized without affecting the substantive content of the present invention.

[0185] like Figure 2 As shown, the device may include, but is not limited to, the following modules:

[0186] (1) Multi-source heterogeneous data acquisition and preprocessing module 210

[0187] This module is used to collect and process multi-source heterogeneous sensing data, including trajectory prefixes, environmental semantic information, road structure, spatial coordinates, weather, and other auxiliary data. It performs data format standardization, outlier removal, noise suppression, interpolation completion, time synchronization, and coordinate transformation to build a stable and consistent input data structure, providing a reliable foundation for subsequent representation learning and conditional modeling.

[0188] (2) Spatiotemporal context representation learning module 220

[0189] This module is used for joint modeling of trajectory dynamics and multimodal environmental information. Through multimodal attention mechanisms, adaptive sequence alignment, trajectory aggregation, and multi-head self-attention structures, it can capture motion trends across time scales and the interdependence between trajectory and environmental semantics, outputting a compact and semantically consistent spatiotemporal context vector, which serves as an important input to conditional generative models.

[0190] (3) Graph network embedding module 230

[0191] This module is used for structured representation of multimodal environmental data. Based on an unlabeled node graph construction strategy, it infers the potential topological relationships between environmental elements through similarity measurement, mutual information calculation, and iterative updates of the adjacency matrix. By optimizing the weights between nodes through graph attention networks and contrastive learning, it obtains noise-robust and topologically stable multimodal graph embedding vectors, providing prior environmental structure for cross-modal modeling.

[0192] (4) Cross-modal attention fusion module 240

[0193] This module is used to fuse trajectory representations and contextual graph embeddings. It employs a dual-channel interactive structure, enhancing the sensitivity of trajectory features to contextual semantics through cross-attention and maintaining continuity and consistency within each modality through self-attention. Ultimately, it generates a unified cross-modal conditional representation to guide subsequent potential trajectory generation.

[0194] (5) Conditional Probability Generation Model Module 250

[0195] Used to generate future trajectories during the inference phase. This module loads a lightweight student model obtained through consistency distillation during the training phase and performs the following functions:

[0196] ① Use a conditional variational autoencoder (CVAE) to map cross-modal conditional vectors to a low-dimensional latent space;

[0197] ② Perform inverse conditional denoising sampling in the latent diffusion model (LDM) to progressively infer the potential representation of future trajectories from random initial noise;

[0198] ③ A binary masking mechanism is used to process missing or sparse trajectory fragments to ensure the structural integrity and spatiotemporal continuity of the generated results.

[0199] It should be noted that this module only performs forward generation during the inference phase and does not include any training, gradient optimization, or distillation operations.

[0200] (6) Trajectory Output Module 260

[0201] This module decodes potential trajectory representations into actual spatial trajectory points and performs necessary rationality checks and consistency adjustments based on road topology constraints, kinematic constraints, and environmental semantic information. It includes steps such as trajectory smoothing, feasibility checks, and semantic filtering, ultimately outputting a future trajectory that satisfies continuity, environmental feasibility, and logical consistency. It can be applied to various scenarios such as ecological monitoring, intelligent traffic management, and unmanned system navigation.

[0202] The aforementioned functional modules can be deployed in a single hardware device, or they can operate collaboratively across servers, mobile terminals, or edge computing nodes. Those skilled in the art can adjust the module division according to actual needs without affecting the scope of protection of this invention.

[0203] VI. Examples of Electronic Equipment

[0204] A third aspect of the present invention also provides an electronic device for performing the trajectory prediction method based on a multimodal conditional latent diffusion generation model described above. This electronic device may be a server, a personal terminal, an embedded system, a mobile computing platform, or any hardware device with computing capabilities.

[0205] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable by the processor. When the processor runs the computer program, it can implement all the steps of the method described in the first aspect of the present invention, including multimodal feature representation learning, graph structure association modeling, cross-modal fusion, conditional diffusion generation, and predicted trajectory output.

[0206] like Figure 3As shown, the electronic device may include, but is not limited to, the following hardware components:

[0207] Processor 1010: can be a central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other executable instruction processing unit, used to run the trajectory prediction model and execute all the functional modules of the present invention.

[0208] Memory 1020 may include ROM, RAM, flash memory, EEPROM, or other non-transitory storage media for storing the operating system, application programs, and computer program instructions for implementing the method of the present invention. When the processor executes these instructions, it can complete all the trajectory prediction steps described in any embodiment of this specification.

[0209] Input / output interface 1030: Used to connect input or output devices to achieve data acquisition and feedback. Input devices may include keyboards, touch screens, sensors, etc.; output devices may include displays, speakers, or other visual / audio output devices.

[0210] Communication interface 1040: Used for communicating with external devices in wired or wireless manner, and may include Ethernet interface, USB interface, cellular mobile communication module, Bluetooth module, Wi-Fi module, etc., to support data transmission and collaborative operation between devices.

[0211] Bus 1050: Used to connect the above hardware components to realize data interaction and control signal transmission between the processor, memory, input / output interface and communication interface.

[0212] It should be understood that Figure 3 The hardware structure shown is merely an example and does not constitute a limitation of the present invention. Depending on the application scenario, the electronic device may add or remove hardware components. For example, it may be equipped with acceleration units such as GPUs, NPUs, and TPUs to improve the efficiency of deep learning inference, or the modules may be recombined to adapt to embedded deployment, cloud deployment, or edge computing environments.

[0213] When the electronic device runs the computer program instructions provided by this invention, it can sequentially execute the processing steps of multimodal perception modeling, latent space condition inference, diffusion generation, distillation-accelerated inference, and trajectory output according to the method steps of this invention, thereby obtaining high-precision and robust future trajectory prediction results in complex, noisy, and highly dynamic environments.

[0214] VII. Examples of Non-Transient Computer-Readable Storage Media

[0215] A fourth aspect of the present invention also provides a non-transitory computer-readable storage medium storing computer program instructions. When the instructions are read and executed by a computer, the computer is able to implement all the steps of the trajectory prediction method based on a multimodal conditional latent diffusion generation model as described in the first aspect of the present invention, including multimodal feature representation learning, graph structure association modeling, cross-modal fusion, conditional diffusion generation, and predicted trajectory output.

[0216] The non-transitory computer-readable storage medium may include, but is not limited to:

[0217] Disks, optical discs, flash memory, solid-state drives (SSDs), phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), or other storage devices capable of storing computer-executable instructions. Any device capable of storing program code in a non-transitory manner, allowing a processor to read and execute it, can be considered a computer-readable medium according to this invention.

[0218] It should be understood that the computer-readable storage medium is not limited to the above-described example types. Any medium capable of storing computer-executable instructions and enabling the computer to run the trajectory prediction method of the present invention is within the scope of protection of the present invention.

[0219] VIII. Summary of Technical Results

[0220] As can be seen from the above embodiments, by constructing a unified trajectory prediction framework based on a multimodal conditional latent diffusion generation model, and with corresponding devices and electronic equipment, the present invention can achieve the following technical effects:

[0221] (1) Track prediction capability that maintains high accuracy and high continuity under noisy, sparse and missing conditions.

[0222] This invention utilizes DTW+K-centroid clustering to aggregate trajectory groups, effectively extracting representative historical motion patterns. By combining multimodal attention, CVAE latent space coding, and LDM conditional diffusion generation, it achieves unified modeling of trajectory uncertainty and randomness. Even with noisy, sparse, or partially missing input data, it can still generate temporally continuous, spatially smooth predicted trajectories that closely approximate real motion patterns, effectively reducing trajectory drift, jump points, and breaks.

[0223] (2) Make full use of environmental structure information to maintain stronger robustness and generalization ability in complex scenarios.

[0224] This invention automatically models the potential structural relationships between environmental topology, spatial neighborhood, and semantic context through an iterative graph network embedding module. Combined with a cross-modal attention mechanism, it enables bidirectional information interaction between trajectory and environment. This allows the system to accurately identify the potential movement trends of targets and generate stable and reliable prediction results in diverse and complex scenarios such as urban roads, woodlands, and integrated indoor and outdoor environments. This cross-modal structural perception condition guidance mechanism gives the prediction model stronger environmental adaptability and generalization ability.

[0225] (3) Lightweighting of the model is achieved through diffusion-consistent distillation, which accelerates the inference process and supports real-time applications.

[0226] This invention employs Diffusion Consistency Distillation (DCD) technology to compress the backsampling process of the latent diffusion model, effectively reducing the number of sampling steps and improving the inference speed of the final student model while maintaining generation quality. This lightweight mechanism enables the trajectory prediction system to be deployed on mobile devices, edge devices, and in high-frequency update scenarios, achieving real-time prediction and rapid response.

[0227] In summary, this invention achieves higher prediction accuracy, stronger structural robustness, and higher real-time performance under multi-source heterogeneous sensing conditions, and is applicable to various practical application scenarios such as ecological monitoring, intelligent transportation, smart city governance, and unmanned system navigation.

[0228] IX. General Instructions and Optional Statements

[0229] For ease of explanation, the device functions are described in modular form in this invention. Those skilled in the art will understand that the specific implementation of the modules can be accomplished through software, hardware, or a combination of both. Multiple modules can also be integrated into a single device or distributed across different devices; any implementation that does not alter the basic concept of this invention falls within its protection scope.

[0230] The technical features of different embodiments in this specification can be arbitrarily combined to form new implementation methods without causing logical conflicts or technical contradictions.

[0231] The power supply circuits, ground connections, conventional interface wiring, IC layouts, and other hardware elements known to those skilled in the art, not shown in the accompanying drawings, are all conventional designs and do not constitute a limitation of this invention.

[0232] The embodiments described in this specification are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, various substitutions, modifications, or equivalent improvements can be made without departing from the spirit and substance of the present invention, and all such modifications and improvements should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A spatiotemporal trajectory prediction method based on a multimodal conditional latent diffusion generation model, characterized in that, include: Acquire multi-source heterogeneous sensing data and preprocess the data to construct a unified input feature space; Spatiotemporal context representation learning is performed on the preprocessed data based on a multimodal attention mechanism to generate a spatiotemporal context vector; Based on iterative graph networks, structural association modeling is performed on the multi-source heterogeneous sensing data to generate graph structure embeddings; The spatiotemporal context vector and the graph structure embedding are deeply fused through a dual-channel cross-modal attention fusion module to generate a cross-modal spatiotemporal interactive representation. The cross-modal spatiotemporal interaction representation is encoded into a low-dimensional latent space based on a conditional variational autoencoder (CVAE) to obtain conditional latent variables; Based on the latent diffusion model (LDM), and using the conditional latent variables as conditions, a potential representation of the future spatiotemporal trajectory is generated from the initial noise through an inverse denoising process; The latent representation is decoded and post-processed to output the final predicted future spatiotemporal trajectory.

2. The method according to claim 1, characterized in that, The spatiotemporal context representation learning based on the multimodal attention mechanism includes: The Dynamic Time Warping (DTW) algorithm is used to calculate the similarity between trajectories, and a clustering algorithm is used to cluster historical trajectories to uncover representative movement patterns. A multi-head self-attention mechanism is used to encode the temporal dynamics of the trajectory to obtain a spatiotemporal representation of the trajectory.

3. The method according to claim 1, characterized in that, The structural association modeling based on iterative graph networks includes: Construct an initial node graph and adjacency matrix based on similarity metrics; The adjacency matrix is ​​iteratively optimized using a graph attention network (GAT) combined with a contrastive learning strategy. Low-dimensional graph structure embedding vectors of nodes are generated based on the optimized adjacency matrix.

4. The method according to claim 1, characterized in that, The feature fusion via a dual-channel cross-modal attention fusion module includes: A cross-attention layer is used, with trajectory features and environmental features as query sequences and key-value sequences, respectively, for bidirectional interactive modeling; A self-attention layer is used to process the trajectory features and environmental features output by the cross-attention layer separately in order to maintain the context consistency within each modality; The processed trajectory features are concatenated with environmental features and then passed through a multilayer perceptron (MLP) to obtain the final cross-modal spatiotemporal interactive representation.

5. The method according to claim 1, characterized in that, Also includes: A binary mask sequence is introduced to identify missing points or incomplete segments in the input trajectory data to enhance the robustness of the model under data missing conditions; In the reverse denoising process of the latent diffusion model (LDM), a resampling technique based on the repaint strategy is adopted to dynamically combine known trajectory data with generated unknown trajectory data in order to improve the spatiotemporal consistency of the predicted trajectory.

6. The method according to any one of claims 1 to 5, characterized in that, It also includes a model distillation stage: After obtaining the teacher model during the training phase, a diffusion consistency distillation (DCD) strategy is adopted to optimize the student model by constructing the consistency loss between the teacher model and the student model in the noise estimation space and the potential diffusion dynamic process. The parameters of the teacher model remain fixed, and the student model, after distillation, can maintain trajectory generation quality comparable to that of the teacher model while reducing the inverse denoising sampling steps.

7. The method according to claim 1, characterized in that, The method further includes: The spatiotemporal awareness loss function is used to perform end-to-end optimization of the model.

8. A spatiotemporal trajectory prediction device based on a multimodal conditional latent diffusion generation model, characterized in that, include: The data acquisition and preprocessing module is used to acquire and preprocess multi-source heterogeneous sensing data; The context representation learning module is used for spatiotemporal context representation learning based on a multimodal attention mechanism; The graph structure association modeling module is used for structural association modeling based on iterative graph networks. A cross-modal attention fusion module is used to fuse spatiotemporal context features with graph structure embeddings through a dual-channel structure; The Conditional Probabilistic Generative Model module is used to generate future trajectories in the latent space based on CVAE and LDM, with cross-modal fusion features as conditions. The trajectory output module is used to decode the generated trajectory, verify its rationality, perform post-processing, and output the final prediction result.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1 to 7.

Images