Trajectory Prediction System and Method Based on Multimodal Thinking Chain

Abstract

This invention relates to the field of trajectory prediction technology, specifically to a trajectory prediction system and method based on a multimodal thinking chain. The system includes an input module, a thinking module, a prediction module, and a training module. These modules collaborate to achieve a decoupled two-stage trajectory prediction paradigm. This application proposes a novel decoupling paradigm that breaks down trajectory prediction into two independent stages: first, a thinking stage, which generates a structured reasoning chain containing key sub-tasks such as destination, direction, and social relationships; second, a prediction stage, which generates the final trajectory based on this reasoning chain. This decoupling design fundamentally solves the goal coupling problem in single-stage models.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction technology, specifically to a trajectory prediction system and method based on multimodal thinking chains. Background Technology

[0002] Pedestrian trajectory prediction is a core technology for systems such as autonomous driving, robot navigation, and intelligent monitoring. Accurately predicting the future trajectory of pedestrians in complex scenes requires not only understanding individual movement patterns but also modeling complex social interactions and environmental dynamics. In recent years, inspired by the powerful reasoning capabilities of Large Language Models (LLMs), methods that reconstruct trajectory prediction as a language task (such as LMTraj) have made significant progress. These methods demonstrate great performance potential by presenting trajectory prediction as a question-answering task. However, these methods have two key limitations:

[0003] First, they use a simple question-and-answer format, lack an explicit chain of reasoning, and are difficult to provide an explainable basis for decision-making;

[0004] Secondly, they mainly rely on coarse-grained image captions for multimodal fusion, losing rich spatial details and interaction cues in the scene, thus limiting their predictive capabilities in complex environments.

[0005] To address the two key issues mentioned above, a readily apparent solution for engineers is to borrow the Chain-of-Thought (CoT) technique from language models, allowing the model to generate intermediate reasoning steps before predicting trajectories, thereby enhancing interpretability. Simultaneously, employing more refined multimodal fusion methods to fully utilize visual information results in a single-stage end-to-end architecture based on a language model, such as... Figure 3 As shown. However, this technical solution of directly introducing CoT into the traditional single-stage end-to-end architecture does not bring the expected performance improvement. On the contrary, it may lead to performance degradation due to the coupling and conflict between the two goals of generating inference and generating trajectories. It can be seen that simply imitating the form of the thought chain is not enough. The key is how to design a framework that allows inference to effectively serve prediction. In addition, although directly using visual features can preserve more spatial details, how to effectively integrate them with language and trajectory information remains a challenge that requires in-depth research. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a trajectory prediction system and method based on multimodal thinking chain.

[0007] The present invention is achieved through the following technical solution: a trajectory prediction system based on a multimodal thinking chain, including an input module, a thinking module, a prediction module and a training module, wherein each module works together to achieve trajectory prediction in a two-stage decoupled paradigm of thinking-prediction;

[0008] The input module is used to receive scene images, target historical observation trajectories, context of neighboring intelligent agents, and question template information;

[0009] The thinking module is used to perform multimodal fusion of information and generate structured reasoning chains;

[0010] The prediction module uses the inference chain as a strong conditional signal and combines it with the information received by the input module to generate the target's future trajectory.

[0011] The training module employs a progressive two-stage training strategy for optimization.

[0012] The thinking module includes an encoder-decoder structure and a multimodal fusion unit;

[0013] The encoder-decoder structure includes a visual encoder, a language encoder, and a language decoder. The visual encoder extracts visual features, and the language encoder converts relevant information into a context-dependent representation.

[0014] The multimodal fusion unit fuses context-related representations with visual features in a multimodal manner to form a multimodal representation;

[0015] The language decoder generates an autoregressive output sequence from the multimodal representation, generates an inference chain during the thinking phase, and generates predicted trajectory coordinates during the prediction phase.

[0016] The multimodal fusion unit employs a multi-head cross-attention mechanism and an adaptive gating mechanism;

[0017] The final output of the multimodal fusion unit is represented by a set of tags enhanced by visual context.

[0018] The visual encoder segments the input image into N×N regions, and each region generates a visual feature vector through linear projection and position encoding.

[0019] The prediction module includes the following two prediction generation modes:

[0020] Deterministic prediction patterns generate the most probable future path. .

[0021] A stochastic prediction model is used to generate K possible multimodal future trajectories with associated confidence levels. k = 1, 2, 3…K.

[0022] The training module uses the cross-entropy loss function. During training, the thinking module is trained first, followed by the prediction module.

[0023] The training module training process includes a thinking phase training and a prediction phase training;

[0024] The thinking phase training primarily focuses on training the generated reasoning chains, which are represented as follows:

[0025] ;

[0026] in, For the true inference chain labeled sequence, The sequence of inference chain markers generated for the thinking module.

[0027] The prediction phase training, followed by the thinking phase training, utilizes real, rule-generated reasoning chains. As a strong condition, the training generates future trajectories, which are represented as follows:

[0028] ;

[0029] in, For the actual trajectory label sequence, In a given true chain of reasoning The trajectory marker sequence generated under the given conditions.

[0030] The trajectory prediction method based on multimodal thinking chains, applied to the aforementioned trajectory prediction system based on multimodal thinking chains, includes the following steps:

[0031] S1. Receive scene images, target historical observation trajectory, context of neighboring agents, and question template information;

[0032] S2, the thinking stage, involves multimodal fusion of information to generate a structured reasoning chain;

[0033] S3. In the prediction stage, the inference chain is used as a strong conditional signal to guide the language decoder to generate trajectory coordinates consistent with the inference logic based on the complete context autoregression.

[0034] The inference chain includes a set of destination predictions. Directional analysis Group Relationship Identification Collision risk assessment Behavioral imitation The intermediate conclusions are expressed as follows:

[0035] ;

[0036] In the formula, This represents a reasoning chain.

[0037] The destination prediction , is defined as the final position coordinates of the target at the end of the prediction time window;

[0038] Directional Analysis This is used to determine the overall direction of movement of the target being analyzed.

[0039] Behavioral imitation , is defined as recognizing the behavioral similarity between a target and other intelligent agents in a scene;

[0040] Group Relationship Identification , is defined as identifying whether a target forms a group with other intelligent agents;

[0041] Collision risk assessment It is defined as assessing the potential collision risk between a target and other intelligent agents.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This application proposes a novel decoupling paradigm that breaks down trajectory prediction into two independent stages: the first is a thinking stage, which generates a structured reasoning chain containing key sub-tasks such as destination, direction, and social relationships; the second is a prediction stage, which generates the final trajectory based on this reasoning chain. This decoupling design fundamentally solves the goal coupling problem in single-stage models.

[0044] This application introduces a dedicated visual encoder to extract fine-grained scene features. Through an advanced multimodal fusion mechanism, it deeply aligns these refined visual features with textual and trajectory information, providing a robust multimodal foundation for generating high-quality inference chains. Furthermore, this design not only addresses the insufficient utilization of visual information in existing methods but also provides rich and uncompressed scene context for the reasoning phase.

[0045] This application employs a rule-based automated method to generate structured, realistic inference chains, avoiding expensive and inconsistent manual annotation, and ensuring the consistency and trainability of the inference chains. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the system architecture of this application;

[0047] Figure 2 This is a schematic diagram of the two-stage decoupling paradigm of this application;

[0048] Figure 3 This is a schematic diagram of an existing single-stage end-to-end architecture based on a language model. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1

[0051] Reference Figure 1 and Figure 2 This embodiment proposes a trajectory prediction system based on a multimodal thinking chain, including an input module, a thinking module, a prediction module and a training module. The modules work together to realize trajectory prediction in a two-stage decoupled paradigm of thinking and prediction.

[0052] The input module is used to receive scene images, target historical observation trajectories, context of neighboring intelligent agents, and question template information;

[0053] The thinking module is used to perform multimodal fusion of information and generate structured reasoning chains;

[0054] The thinking module includes an encoder-decoder structure and a multimodal fusion unit;

[0055] The encoder-decoder structure includes a visual encoder, a language encoder, and a language decoder. The visual encoder extracts visual features, and the language encoder converts relevant information into context-sensitive representations.

[0056] The multimodal fusion unit performs multimodal fusion of the context-related representation converted by the language encoder and the visual features extracted by the visual encoder to form a rich multimodal representation;

[0057] The language decoder generates an output sequence from the multimodal representation via autoregression.

[0058] During the thinking phase, the language encoder receives relevant information including the target's historical observation trajectory, the context of neighboring agents, and question template information;

[0059] During the prediction phase, the language encoder receives relevant information including the target's historical observation trajectory, the context of neighboring agents, question template information, and the inference chain output from the thinking phase.

[0060] The visual encoder segments the input image into N×N regions, and each region generates a feature vector through linear projection and positional encoding. This application uses VisionTransformer (ViT) as the visual feature extractor (i.e., visual encoder). Compared with global features, this application can better preserve the spatial structure and local details of the image, providing rich visual context for subsequent multimodal fusion.

[0061] The multimodal fusion unit is based on a multi-head cross-attention mechanism to achieve deep fusion of visual features and text / trajectory. Specifically, in this embodiment, text tags are used as Q (Query), visual feature vectors are used as K (Key) and V (Value), and attention weights are calculated. The calculation formula is as follows:

[0062] ;

[0063] in, The dimension of the Key.

[0064] To balance the importance of different modal information, the multimodal fusion unit in this embodiment also introduces an adaptive gating mechanism. It is represented as follows:

[0065] ;

[0066] in, The context-dependent representation of the language encoder's transformation is represented by the gating weights, which are calculated using a learnable function, as follows:

[0067] ;

[0068] This represents the sigmoid activation function. and These are learnable parameters.

[0069] The final output of the multimodal fusion unit is represented by a set of visual context-enhanced tags, each tag incorporating relevant visual feature information. These fused features are then fed into the language decoder for autoregressive decoding.

[0070] In summary, the relevant information received by the language encoder is converted into context-sensitive representations, which are then fused with visual features extracted by the visual encoder through a multimodal fusion unit to form rich multimodal representations. The language decoder generates output sequences based on multimodal autoregression, specifically generating inference chains and trajectory coordinates during the thinking and prediction phases, respectively.

[0071] Based on the language decoder, the thinking module outputs a natural language text representation, which makes the decision-making process completely transparent and interpretable.

[0072] The prediction module uses the inference chain as a strong conditional signal and combines it with the information received by the input module to generate the target's future trajectory.

[0073] This application supports multiple inference strategies, which can generate both deterministic optimal predictions and diverse multimodal prediction results to meet the needs of different application scenarios.

[0074] Furthermore, the prediction module is divided into two prediction generation modes based on task requirements, specifically including:

[0075] Deterministic prediction patterns generate the most probable future path. In this mode, a beam search strategy is used to generate the inference chain with the highest probability, and the most likely trajectory is generated based on this inference chain. This strategy is suitable for scenarios requiring stable and repeatable predictions.

[0076] A stochastic prediction model is used to generate K possible multimodal future trajectories with associated confidence levels. For each k=1, 2, 3…K, a fixed-thinking, divergent-action strategy is employed in this mode to generate diverse prediction results. The thinking module generates a high-quality (fixed) inference chain through bundle search, while the prediction module generates multiple different trajectories using temperature sampling, constrained by the fixed inference chain. This strategy ensures both consistency of inference and diversity of predictions.

[0077] The training module employs a progressive two-stage training strategy for optimization.

[0078] The training module uses the cross-entropy loss function. During training, the thinking module is trained first, followed by the prediction module. This design effectively avoids interference between tasks, thus ensuring full optimization of each subtask.

[0079] Specifically, the training module training process includes two stages: the thinking stage training and the prediction stage training.

[0080] For the thinking phase training, the main focus is on training the generated reasoning chains, which are represented as follows:

[0081] ;

[0082] in, For the true inference chain labeled sequence, The sequence of inference chain markers generated for the thinking module.

[0083] For the prediction phase training, after the thinking phase training is completed, a real, rule-generated inference chain is used. As a strong condition, the training generates future trajectories, which are represented as follows:

[0084] ;

[0085] in, For the actual trajectory label sequence, In a given true chain of reasoning The trajectory marker sequence generated under the given conditions.

[0086] Traditional multimodal fusion methods often convert visual information into simple image descriptions and global features, losing fine-grained spatial details and local context. This application directly extracts raw image features through a visual encoder and achieves early deep fusion of visual, trajectory, and textual information through a carefully designed multimodal fusion unit, providing rich multimodal representations for the first-stage thought chain generation and the second-stage trajectory prediction.

[0087] Example 2

[0088] Based on Example 1, combined with Figure 1 and Figure 2 As shown in the figure, this embodiment provides a trajectory prediction method based on multimodal thinking chain.

[0089] This embodiment restates the target trajectory prediction problem. Specifically, for each target... Given scene image Textualized historical observation trajectory Context of neighboring agents Question Template Its ultimate goal is to generate its future trajectory. Based on this, this embodiment decouples the prediction process into two stages: thinking and prediction.

[0090] The thinking stage is represented as follows:

[0091] ;

[0092] The prediction phase is represented as:

[0093] .

[0094] in, It is a structured reasoning chain.

[0095] The trajectory prediction method based on multimodal thinking chain described in this embodiment includes the following steps:

[0096] S1. Receive scene images, target historical observation trajectory, context of neighboring agents, and question template information;

[0097] S2, the thinking phase, involves multimodal information fusion, with the language decoder generating a structured reasoning chain in an autoregressive manner. The inference chain Includes predefined answers to auxiliary tasks.

[0098] The answers to the auxiliary tasks include destination prediction, direction analysis, group relationship identification, collision risk assessment, and behavior imitation.

[0099] In other words, the inference chain Includes a set of destination predictions Directional analysis Group Relationship Identification Collision risk assessment Behavioral imitation The intermediate conclusions of key subtasks are represented as follows:

[0100] .

[0101] This embodiment selects five core tasks as the basis for the structured reasoning chain. Each task is equipped with a specific text template to describe its content, including:

[0102] Destination prediction is defined as the final position coordinates of the target at the end of the prediction time window. Its template is: target_n_ will be in the following... The frame arrives at coordinates (x, y).

[0103] Directional Analysis This is used to determine the overall direction of movement of the target being analyzed.

[0104] Its template is: target_n_direction. Where direction can be forward, backward, left, right, or stop, etc.

[0105] Behavioral imitation It is defined as recognizing the similarity of the behavior of a target with that of other intelligent agents in the scene.

[0106] Its template is: the target _n_ walks in a similar way to the agent (such as a pedestrian) _k_, or the target _n_ will walk alone.

[0107] Group Relationship Identification , is defined as identifying whether a target forms a group with other intelligent agents.

[0108] Its template is: target_n_ and agent_k_ form a group, or target_n_ will walk alone.

[0109] Collision risk assessment It is defined as assessing the potential collision risk between a target and other intelligent agents.

[0110] Its template is: there is a risk of collision between target _n_ and agent _k_, or there is no risk of collision between target _n_.

[0111] After generating specific text, the above task concatenates them into a single string, thus forming a complete and structured reasoning chain. An example of a reasoning chain is as follows:

[0112] Target 0 will arrive at coordinates (153, 113) in the next 12 frames. Target 0 will move forward. Target 0's movement pattern is similar to that of pedestrian 1. Target 0 and pedestrian 1 form a group. There is a risk of collision between target 0 and pedestrian 1.

[0113] This application employs a predefined structured format for its inference chain, ensuring consistency and parsability. Each task's text has explicit identifiers and delimiters, facilitating subsequent parsing and processing. This systematic, rule-based approach to constructing realistic thought chains ensures consistency, directly reflects the underlying real data, and provides clear, structured supervision for the training and prediction phases.

[0114] Furthermore, the structured reasoning chain generation method of this application can be extended to new auxiliary tasks, requiring only the definition of corresponding templates and generation rules. This modular design gives the system good scalability and adaptability.

[0115] S3, Prediction Phase, using the inference chain As a strong conditional signal, it guides the language decoder to generate trajectory coordinates consistent with the inference logic based on autoregression of the complete context.

[0116] The above description is merely an optional embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the content of the present invention under the concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A trajectory prediction system based on multi-modal thought chain, characterized in that, The system comprises an input module, a thinking module, a prediction module and a training module, and the modules are cooperated to realize the thinking-prediction two-stage decoupling paradigm trajectory prediction. The input module is configured to receive scene images, target historical observation trajectories, neighboring agent contexts and problem template information. The thinking module is configured to perform multi-modal fusion of information and generate a structured reasoning chain. The prediction module generates the future trajectory of the target by taking the reasoning chain as a strong conditional signal and combining the information received by the input module. The training module adopts a progressive two-stage training strategy for training optimization. The training module adopts a cross-entropy loss function Training, in the training, first train the thinking module, then train the prediction module; The training process of the training module includes thinking stage training and prediction stage training. The thinking stage training includes training the reasoning chain generated by the thinking module, which is represented as follows: ； wherein, is a sequence of reasoning chain tags that are true, is a sequence of reasoning chain tags generated by the thinking module; The prediction stage training uses real reasoning chains as strong conditions to train the generation of future trajectories, which is represented as follows: ； wherein, is a sequence of trajectory labels that is true, is a sequence of trajectory labels generated under the condition that the given true chain of inferences is true.

2. The multi-modal thought chain based trajectory prediction system of claim 1, wherein, The thinking module comprises an encoder-decoder structure and a multi-modal fusion unit. The encoder-decoder structure comprises a visual encoder, a language encoder and a language decoder. The visual encoder performs visual feature extraction. The language encoder converts related information into context-related representations.

3. The multi-modal train of thought chain based trajectory prediction system of claim 2, wherein, The multi-modal fusion unit performs multi-modal fusion of the context-related representations and the visual features to form multi-modal representations. The language decoder autoregressively generates output sequences from the multi-modal representations, generates reasoning chains in the thinking stage, and generates predicted trajectory coordinates in the prediction stage.

4. The multi-modal thought chain based trajectory prediction system of claim 2, wherein, The multi-modal fusion unit adopts a multi-head cross-attention mechanism and an adaptive gating mechanism.

5. The multi-modal thought chain based trajectory prediction system of claim 1, wherein, The final output of the multi-modal fusion unit is a set of labeled representations enhanced by visual context. Deterministic prediction mode, generating one most likely future path ; Randomness prediction mode, generating K multimodal future trajectories , k = 1, 2, 3... K.

6. A trajectory prediction method based on multi-modal thought chain, characterized in that, The visual encoder divides the input image into N×N regions, and generates a visual feature vector for each region through linear projection and position encoding. The prediction module includes the following two prediction generation modes: The trajectory prediction system based on multi-modal thinking chain according to any one of claims 1-5 comprises the following steps: The inference chain comprises a set of intermediate conclusions about destination prediction , direction analysis , group relationship identification , collision risk assessment and behavior imitation , which are represented as follows: ； In the formulae, denotes a reasoning chain; S1, receiving scene images, target historical observation trajectories, neighboring agent contexts and problem template information; S2, in the thinking stage, performing multi-modal fusion of information to generate a structured reasoning chain; S3, in the prediction stage, taking the reasoning chain as a strong conditional signal to generate trajectory coordinates consistent with the reasoning logic.

7. The multi-modal thought chain based trajectory prediction method of claim 6, wherein, The destination prediction defined as the final position coordinates of the prediction target at the end of the prediction time window; direction analysis positioning as a whole to analyze the moving direction of the target; Behavioral imitation is defined as recognizing the similarity of the behavior of the target and other agents in the scene; Group relationship identification defined as identifying whether the target forms a group with other agents; Collision risk assessment is defined as assessing the potential collision risk of the target with other agents.

Images