Travel time estimation method based on multi-modal trajectory fusion

Abstract

The invention relates to the field of intelligent traffic, in particular to a travel time estimation method based on multi-modal trajectory fusion, which comprises the following steps: designing three encoders, including a grid encoder, a GPS encoder and a road encoder; an alignment fusion module is introduced, dynamic alignment enhancement of a GPS-grid and multi-modal fusion interaction of the grid after road-enhancement are achieved, and unified multi-modal embedding representation is obtained; optimizing by adopting a dual-objective joint loss function, including a cross-modal contrast learning loss function and a road sequence mask language model loss function until the network converges; and inputting the coded unified multi-modal embedding representation into a TTE regression head to obtain a prediction result. The method has the advantages that multi-mode coding is adopted, and effective fusion of heterogeneous multi-source track data is achieved; the track information density is adjusted in a self-adaptive mode, and the stable representation effect is guaranteed; two complementary loss functions are applied to stabilize training and optimize performance, and travel time estimation precision and robustness are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation, and in particular to a method for travel time estimation based on multimodal trajectory fusion. Background Technology

[0002] With the rapid development of connected vehicle technology, travel time estimation has become a core task in traffic flow analysis, primarily predicting vehicle travel times on different road segments by analyzing traffic data. The main challenge for existing technologies lies in effectively modeling the inherent complexity of multimodal spatiotemporal traffic systems. In recent years, researchers have relied on single-source trajectory data to estimate travel times; however, discrete GPS trajectory data is often affected by drift, grid-based road network modeling has limited accuracy, and road-based analysis largely depends on real-time updates. Therefore, relying on single-source trajectory data representation is insufficient for accurate and reliable travel time estimation. Furthermore, many existing methods address trajectory variability through representation learning by leveraging the strengths of different sources. However, these strategies often introduce redundant noise or lead to information loss.

[0003] Existing research focuses on the following two questions: (i) how to extract semantic structures from different modalities and achieve effective alignment and fusion; (ii) how to incorporate dynamic modeling into trajectory representation learning to ensure that embeddings of different lengths better capture basic spatiotemporal patterns. Summary of the Invention

[0004] To overcome the above-mentioned shortcomings, the present invention aims to provide a travel time estimation method based on multimodal trajectory fusion, which solves the problems of loss of representation information, feature redundancy and cross-modal semantic inconsistency caused by heterogeneous multi-source trajectory data and variable actual trajectory length.

[0005] This invention achieves the above objective through the following scheme: a travel time estimation method based on multimodal trajectory fusion, comprising the following steps:

[0006] (1) Design three encoders, including:

[0007] (1.1) A grid encoder based on learning grid spatiotemporal embedding using local subgraphs and graph attention modules;

[0008] (1.2) Discretize the continuous coordinates into semantic prototype tags and construct text prompts, and obtain the semantic representation of the GPS encoder with the help of a large language model;

[0009] (1.3) The road graph is updated with a graph attention network to represent nodes, and a road encoder with sequence dependencies is modeled with a Transformer;

[0010] (2) An alignment and fusion module is introduced to realize dynamic alignment enhancement of GPS-mesh and multimodal fusion interaction of road-enhanced mesh, so as to obtain a unified multimodal embedding representation;

[0011] (3) A dual-objective joint loss function is used for optimization, including a cross-modal contrastive learning loss function and a road sequence mask language model loss function, until the network converges;

[0012] (4) Input the encoded unified multimodal embedding representation into the TTE regression head to obtain the prediction results.

[0013] Preferably, the mesh encoder design in (1.1) specifically includes the following steps:

[0014] (1.1.1) Discretize the study area into an M×N grid, construct a 3×3 local subgraph for each grid cell according to the 8-neighborhood rule, form a first-order adjacency matrix A, and retain only the connection between itself and its eight neighboring nodes;

[0015] (1.1.2) Using the channel vectors of each grid as the initial node features, a graph attention network is used to aggregate neighborhood information and update node embeddings. After stacking several layers, a sequence representation of dimension T×D is obtained through linear projection. ,in The [CLS] marker represents the mesh embeddings from different time periods; the mesh sequence output by the mesh encoder represents... It is then fed into the Transformer model as input for sequence encoding.

[0016] Preferably, the GPS encoder design in (1.2) specifically includes the following steps:

[0017] (1.2.1) Each trajectory is decomposed into a series of structured text fragments with clear semantics, wherein each trajectory is represented as a time step sequence T, and each time step sequence T consists of three digital channels of corresponding longitude, latitude and time features. These channels together describe the spatiotemporal state of the moving object at each sampling point and serve as the basic input for local motion feature extraction.

[0018] (1.2.2) Divide the trajectory into non-overlapping blocks of length three to generate a set of sub-trajectories. ;

[0019] (1.2.3) Using time index, observed values, and cumulative trends Matched prototype identifiers and similarity integration Construct text prompts, Used to bridge the modal differences between continuous coordinates and discrete text, where For each sub-trajectory The flattened 9-dimensional vector representation, This represents the pattern library, where K represents the number of predefined reference patterns. This represents the motion value observed at timestamp t;

[0020] (1.2.4) will The text is fed into a large language model, and the hidden state of the last token is taken as the GPS sequence representation of the text segment. .

[0021] Preferably, the road encoder design in (1.3) specifically includes the following steps:

[0022] (1.3.1) The road segment node features are linearly projected onto a unified dimension, and the convolutional layers of the stacked graph attention network update the node embeddings according to the adjacency relationship N(i) of the road network topology to obtain the road perception graph embedding; based on this embedding, three special labels [PAD], [CLS], and [MASK] are introduced, and temporal embedding, type embedding and location encoding PE(·) are superimposed; a padding mask is used to apply attention only to the effective road segments;

[0023] (1.3.2) In each Transformer model layer, self-attention is first performed on the type embedding to obtain the prior B, which is then injected into the self-attention calculation. ,in, These represent the query, key, and value matrices obtained from the previous hidden representation after a linear transformation. Let be the dimension of the key vector. The prior attention bias matrix is ​​obtained from type embedding self-attention. This indicates that each row is normalized to obtain attention weights;

[0024] (1.3.3) After passing through multiple Transformer models, the output sequence representation is projected into the shared space. The first [CLS] token is used as the trajectory-level representation, representing the encoded road sequence as follows: .

[0025] Preferably, step (2) specifically includes the following steps:

[0026] (2.1) Represent the GPS sequence output by the GPS encoder Projected into the latent space consistent with the mesh encoder: ;

[0027] (2.2) A comparison strategy is introduced to ensure consistent mapping between GPS sequence representation and grid sequence representation: when the length of GPS sequence representation is greater than that of grid sequence representation, redundant time information is truncated and removed; when the length of GPS sequence representation is less than that of grid sequence representation, a zero-padding strategy is used to maintain a one-to-one correspondence.

[0028] (2.3) Representing the aligned GPS sequence through a residual fusion mechanism Injected into the mesh sequence representation to obtain the enhanced mesh sequence representation ;

[0029] (2.4) Calculate the road sequence representation With enhanced grid sequence representation Cross-channel precautions: in For a learnable projection matrix, Indicates spatial alignment offset;

[0030] (2.5) The fused output is refined through residual connections and a feedforward network to obtain a unified multimodal embedding representation. .

[0031] Preferably, step (3) specifically includes the following steps:

[0032] (3.1) In each trajectory sample, the enhanced grid sequence representation and the road sequence representation are subjected to InfoNCE bidirectional loss calculation with temperature scaling to obtain the cross-modal contrastive learning loss function. ;

[0033] (3.2) Represent the road sequence by randomly masking the locations of 15% of road segments, and calculate the loss function of the road sequence masking language model. , where M represents the set of occluded locations, Indicates that, given a masked sequence Its corresponding enhanced grid sequence representation and road sequence representation Under these conditions, correctly predict road segments The probability of;

[0034] (3.3) Obtain the joint loss function of the two objectives The function is then optimized until the network converges.

[0035] Preferably, in the cross-channel attention calculation, the road sequence representation R is used as Q to actively retrieve relevant contextual information, and the enhanced grid sequence representation... As K and V, they provide rich spatial semantics.

[0036] Preferably, the network encoder contains two Transformer layers.

[0037] Preferably, the time embedding includes a week index and a minute index.

[0038] Preferably, the large language model is a pre-trained language model GPT-2.

[0039] The beneficial effects of this invention are as follows: It employs multimodal coding to extract corresponding semantic structures from different modalities, achieving effective fusion of heterogeneous multi-source trajectory data and reducing information loss and redundancy during the representation process; it can adaptively adjust trajectory information density to reduce redundancy and suppress local embedding bias when trajectory length and sampling density change, ensuring stable representation effects for both long and short trajectories and improving the accuracy of downstream TTE tasks; simultaneously, it enhances semantic consistency and contextual modeling capabilities across modal channels, applying two complementary loss functions to stabilize training and optimize performance, thereby effectively overcoming the shortcomings of existing technologies in terms of travel time estimation accuracy and robustness, and exhibiting stronger generalization and stability. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the steps of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of the network framework of the method of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to specific implementation examples, but the scope of protection of the present invention is not limited thereto:

[0043] Example: Figure 1 , Figure 2 As shown, a travel time estimation method based on multimodal trajectory fusion includes the following steps:

[0044] (1) Three encoders were designed, including a network encoder, a GPS encoder, and a road encoder, to provide preliminary multimodal processing capabilities. Specifically, graph attention and graph neural networks were used to encode road trajectories for map matching, while semantic analysis of a large language model enhanced the representation of the raw GPS data. Specifically:

[0045] (1.1) A grid encoder based on learning grid spatiotemporal embedding of local subgraphs and graph attention modules.

[0046] (1.1.1) Discretize the study area into a regular grid of 100×100. The size of each grid is set to grid_size=100. The spatial dimensions of the grid image are set to height grid_H=114 and width grid_W=52. Construct a 3×3 local subgraph for each grid cell according to the 8-neighbor rule to form a first-order adjacency matrix A. Only retain the connection between itself and eight neighboring nodes to reduce the computational power and memory overhead of global graph construction.

[0047] (1.1.2) Using the channel vectors of each grid as the initial node features, a graph attention network is used to aggregate neighborhood information and update node embeddings. After stacking several layers, a sequence representation of dimension T×D is obtained through linear projection. ,in The [CLS] marker represents the mesh embeddings from different time periods; the mesh sequence output by the mesh encoder represents... The input is then fed into a Transformer model for sequence encoding. In this example, the grid feature input has 3 channels, which are linearly transformed to 64 dimensions. This is then fed into a grid encoder containing two Transformer layers, with 4 multi-head attention heads, a dropout rate of 0.1, and a feedforward network dimension of 1024. The output dimension of the grid encoder is 64.

[0048] (1.2) Discretize the continuous coordinates into semantic prototype tags and construct text prompts, and obtain the semantic representation of the GPS encoder with the help of a large language model.

[0049] (1.2.1) Each trajectory is decomposed into a series of structured text fragments with clear semantics, wherein each trajectory is represented as a time step sequence T, and each time step sequence T consists of three digital channels of corresponding longitude, latitude and time features. These channels together describe the spatiotemporal state of the moving object at each sampling point and serve as the basic input for local motion feature extraction.

[0050] (1.2.2) Divide the trajectory into non-overlapping blocks of length three to generate a set of sub-trajectories. ;

[0051] (1.2.3) Each sub-trajectory Flattened into a 9-dimensional vector To capture representative spatiotemporal motion patterns, we will... With pattern library Compare and calculate the similarity between the two. This formula, by exponentially applying the Euclidean distance, transforms the similarity between each sub-trajectory and the prototype into comparable weights, thereby identifying the most representative motion patterns. To further enhance semantic expressiveness, we also extract cumulative trend values ​​from the original sequence: Using time index, observed values, and cumulative trends Matched prototype identifiers and similarity integration Construct text prompts, Used to bridge the modal differences between continuous coordinates and discrete text, where K represents the number of predefined reference modes. This represents the motion value observed at timestamp t;

[0052] (1.2.4) will The text is fed into a pre-trained large language model GPT-2, and the hidden state of the last token is taken as the GPS sequence representation of the text segment. .

[0053] (1.3) The road graph is represented by a graph attention network to update nodes, and the sequential dependency road encoder is modeled with a Transformer.

[0054] (1.3.1) The road segment node features are linearly projected onto a unified dimension, and the convolutional layers of the stacked graph attention network update the node embeddings according to the adjacency relationship N(i) of the road network topology to obtain the road perception graph embedding; based on this embedding, three special labels [PAD], [CLS], and [MASK] are introduced, and time embedding (including week index and minute index), type embedding and location encoding PE(·) are superimposed; a padding mask is used to apply attention only to valid road segments;

[0055] (1.3.2) In each Transformer model layer, self-attention is first performed on the type embedding to obtain the prior B, which is then injected into the self-attention calculation. .in, These represent the query, key, and value matrices obtained by linear transformation of the previous hidden representation. Let be the dimension of the key vector. The prior attention bias matrix is ​​obtained from type embedding self-attention. This indicates that each row is normalized to obtain attention weights;

[0056] (1.3.3) After passing through multiple Transformer layers, the output sequence representation is projected into the shared space. The first [CLS] token is used as the trajectory-level representation, representing the encoded road sequence as follows: .

[0057] (2) An alignment and fusion module is introduced to achieve dynamic alignment enhancement of GPS-mesh and multimodal fusion interaction between roads and the enhanced mesh, resulting in a unified multimodal embedding representation. Specifically:

[0058] (2.1) Represent the GPS sequence output by the GPS encoder Projected into the latent space consistent with the mesh encoder: ;

[0059] (2.2) Regarding length, a comparison strategy is introduced to ensure consistent mapping between GPS sequence representation and grid sequence representation: when the length of GPS sequence representation is greater than that of grid sequence representation, redundant time information is truncated and removed; when the length of GPS sequence representation is less than that of grid sequence representation, a zero-padding strategy is used to maintain a one-to-one correspondence; this mechanism achieves precise time step synchronization between channels and ensures time consistency between GPS signals and grid markers, providing a solid representation foundation for subsequent fusion.

[0060] (2.3) Representing the aligned GPS sequence through a residual fusion mechanism Injected into the mesh sequence representation to obtain the enhanced mesh sequence representation ;

[0061] (2.4) Calculate the road sequence representation With enhanced grid sequence representation Cross-channel precautions:

[0062]

[0063] in For a learnable projection matrix, This represents spatial alignment offset; in this design, the road sequence R acts as a query, actively retrieving relevant contextual information, while the fused mesh representation... As key and value pairs, it provides rich spatial semantics. This form enables adaptive, context-aware feature interactions between different modalities, guided by the trajectory structure.

[0064] (2.5) The fused output is refined through residual connections and a feedforward network to obtain a unified multimodal embedding representation. .

[0065] (3) A dual-objective joint loss function is used for optimization, including a cross-modal contrastive learning loss function and a road sequence masking language model loss function, until the network converges. Specifically:

[0066] (3.1) In each trajectory sample, the enhanced grid sequence representation and the road sequence representation are subjected to InfoNCE bidirectional loss calculation with temperature scaling to obtain the cross-modal contrastive learning loss function. ;

[0067] (3.2) Represent the road sequence by randomly masking the locations of 15% of road segments, and calculate the loss function of the road sequence masking language model. , where M represents the set of occluded locations, Indicates that, given a masked sequence Its corresponding enhanced grid sequence representation and road sequence representation Under these conditions, correctly predict road segments The probability of;

[0068] (3.3) Obtain the final self-supervised objective: the bi-objective joint loss function The function is optimized until the network converges. The two losses are weighted equally, and cross-modal alignment and intra-modal temporal dependencies are learned simultaneously. In this example, the multimodal interaction module is set to a 2-layer Transformer with 2 attention heads, a dropout rate of 0.1, and a feedforward network dimension of inter_ffn_dim=4×out_emb_dim=1024. The weighting coefficient lambda_cross_modal_relation for cross-modal relationship modeling is set to 0.5, with weights of step consistency, relational consistency, and cosine consistency of 1.0, 1.0, and 1.0, respectively.

[0069] (4) Input the encoded unified multimodal embedding representation into the TTE regression head to obtain the prediction results.

[0070] Regarding training hyperparameters, this embodiment uses a batch size of 32, 30 training epochs, and an initial learning rate of 32. The weight decay coefficient is set to The optimizer uses AdamW and combines it with a cosine annealing learning rate adjustment strategy, performing a warmup in the first 10 epochs with an initial learning rate of... The minimum learning rate is the same. The model employs an early stopping mechanism, halting training when the validation set metrics show no improvement for five consecutive rounds. The model's latent representation dimension is 256, maintaining consistency with parameters such as positional encoding dropout (0.1) and output embedding dimension (256).

[0071] This invention employs multimodal fusion to align and integrate information from different modalities, generating a unified embedding representation for travel time estimation. Furthermore, a dynamic trajectory alignment module is introduced, adaptively adjusting the embedding information density based on trajectory length and sampling density to suppress local embedding bias, reduce representation information loss and feature redundancy, and achieve training process stability and overall performance optimization through two complementary loss functions. This enables multi-source constraint fusion and robust trajectory representation in complex traffic dynamic scenarios. The method achieves three metrics (MAE, RMSE, MAPE) of 1.419 / 2.5347 / 0.152, 1.1693 / 1.6144 / 0.1701, and 1.1439 / 1.5077 / 0.1838 on the Porto, Xian, and Chengdu datasets, respectively.

[0072] The above description describes specific embodiments of the present invention and the technical principles employed. Any changes made in accordance with the concept of the present invention that do not exceed the spirit of the specification and drawings should still fall within the protection scope of the present invention.

Claims

1. A travel time estimation method based on multimodal trajectory fusion, characterized in that... Includes the following steps: (1) Design three encoders, including: (1.1) A grid encoder based on learning grid spatiotemporal embedding using local subgraphs and graph attention modules; (1.2) Discretize the continuous coordinates into semantic prototype tags and construct text prompts, and obtain the semantic representation of the GPS encoder with the help of a large language model; (1.3) The road graph is updated with a graph attention network to represent nodes, and a road encoder with sequence dependencies is modeled with a Transformer; (2) An alignment and fusion module is introduced to realize dynamic alignment enhancement of GPS-mesh and multimodal fusion interaction of road-enhanced mesh, so as to obtain a unified multimodal embedding representation; (3) A dual-objective joint loss function is used for optimization, including a cross-modal contrastive learning loss function and a road sequence mask language model loss function, until the network converges; (4) Input the encoded unified multimodal embedding representation into the TTE regression head to obtain the prediction results.

2. The travel time estimation method based on multimodal trajectory fusion according to claim 1, characterized in that, The mesh encoder design described in (1.1) specifically includes the following steps: (1.1.1) Discretize the study area into an M×N grid, construct a 3×3 local subgraph for each grid cell according to the 8-neighborhood rule, form a first-order adjacency matrix A, and retain only the connection between itself and its eight neighboring nodes; (1.1.2) Using the channel vectors of each grid as the initial node features, a graph attention network is used to aggregate neighborhood information and update node embeddings. After stacking several layers, a sequence representation of dimension T×D is obtained through linear projection. ,in The [CLS] marker represents the mesh embeddings from different time periods; the mesh sequence output by the mesh encoder represents... It is then fed into the Transformer model as input for sequence encoding.

3. The travel time estimation method based on multimodal trajectory fusion according to claim 1, characterized in that, The GPS encoder design described in (1.2) specifically includes the following steps: (1.2.1) Each trajectory is decomposed into a series of structured text fragments with clear semantics, wherein each trajectory is represented as a time step sequence T, and each time step sequence T consists of three digital channels of corresponding longitude, latitude and time features. These channels together describe the spatiotemporal state of the moving object at each sampling point and serve as the basic input for local motion feature extraction. (1.2.2) Divide the trajectory into non-overlapping blocks of length three to generate a set of sub-trajectories. ; (1.2.3) Using time index, observed values, and cumulative trends Matched prototype identifiers and similarity integration Construct text prompts, Used to bridge the modal differences between continuous coordinates and discrete text, where For each sub-trajectory The flattened 9-dimensional vector representation, This represents the pattern library, where K represents the number of predefined reference patterns. This represents the motion value observed at timestamp t; (1.2.4) will The text is fed into a large language model, and the hidden state of the last token is taken as the GPS sequence representation of the text segment. .

4. The travel time estimation method based on multimodal trajectory fusion according to claim 1, characterized in that, The road encoder design described in (1.3) specifically includes the following steps: (1.3.1) The road segment node features are linearly projected onto a unified dimension, and the convolutional layers of the stacked graph attention network update the node embeddings according to the adjacency relationship N(i) of the road network topology to obtain the road perception graph embedding; based on this embedding, three special labels [PAD], [CLS], and [MASK] are introduced, and temporal embedding, type embedding and location encoding PE(·) are superimposed; a padding mask is used to apply attention only to the effective road segments; (1.3.2) In each Transformer model layer, self-attention is first performed on the type embedding to obtain the prior B, which is then injected into the self-attention calculation. ,in, These represent the query, key, and value matrices obtained from the previous hidden representation after a linear transformation. Let be the dimension of the key vector. The prior attention bias matrix is ​​obtained from type embedding self-attention. This indicates that each row is normalized to obtain attention weights; (1.3.3) After passing through multiple Transformer models, the output sequence representation is projected into the shared space. The first [CLS] token is used as the trajectory-level representation, representing the encoded road sequence as follows: .

5. The travel time estimation method based on multimodal trajectory fusion according to claim 1, characterized in that, Step (2) specifically includes the following steps: (2.1) Represent the GPS sequence output by the GPS encoder Projected into the latent space consistent with the mesh encoder: ; (2.2) A comparison strategy is introduced to ensure consistent mapping between GPS sequence representation and grid sequence representation: when the length of GPS sequence representation is greater than that of grid sequence representation, redundant time information is truncated and removed; when the length of GPS sequence representation is less than that of grid sequence representation, a zero-padding strategy is used to maintain a one-to-one correspondence. (2.3) Representing the aligned GPS sequence through a residual fusion mechanism Injected into the mesh sequence representation to obtain the enhanced mesh sequence representation ; (2.4) Calculate the road sequence representation With enhanced grid sequence representation Cross-channel precautions: in The projection matrix is ​​learnable. Indicates spatial alignment offset; (2.5) The fused output is refined through residual connections and a feedforward network to obtain a unified multimodal embedding representation. .

6. The travel time estimation method based on multimodal trajectory fusion according to claim 1, characterized in that, Step (3) specifically includes the following steps: (3.1) In each trajectory sample, the enhanced grid sequence representation and the road sequence representation are subjected to InfoNCE bidirectional loss calculation with temperature scaling to obtain the cross-modal contrastive learning loss function. ; (3.2) Represent the road sequence by randomly masking the locations of 15% of road segments, and calculate the loss function of the road sequence masking language model. , where M represents the set of occluded locations, Indicates that, given a masked sequence Its corresponding enhanced grid sequence representation and road sequence representation Under these conditions, correctly predict road segments The probability of; (3.3) Obtain the joint loss function of the two objectives The function is then optimized until the network converges.

7. The travel time estimation method based on multimodal trajectory fusion according to claim 5, characterized in that, In the cross-channel attention calculation, the road sequence representation R serves as Q to actively retrieve relevant contextual information, and the enhanced grid sequence representation... As K and V, they provide rich spatial semantics.

8. The travel time estimation method based on multimodal trajectory fusion according to claim 2, characterized in that, The network encoder contains two layers of Transformer.

9. The travel time estimation method based on multimodal trajectory fusion according to claim 4, characterized in that, The time embedding includes a week index and a minute index.

10. A travel time estimation method based on multimodal trajectory fusion according to any one of claims 1 or 3, characterized in that, The large language model is the pre-trained language model GPT-2.

Images