Cross-city traffic prediction method and system based on multi-modal fusion and spatial expert routing

Abstract

The invention discloses a cross-city traffic prediction method and system based on multi-modal fusion and spatial expert routing, relates to the technical field of traffic prediction, and provides an adaptive modal selection mechanism based on a signal-to-noise ratio for the problems of missing, noise and uneven quality of multi-modal data in different cities. And a low-quality mode is dynamically suppressed in combination with comparative learning, and robust multi-mode fusion is realized. In order to solve the problems of large space structure difference and weak generalization ability among cities, a multi-modal guided space expert routing architecture is designed: modal sharing experts and routing experts are activated by using a multi-modal context, and local space dependence is adaptively modeled for different functional regions. The method supports any modal combination input, city specific fine tuning is not needed, and the zero sample cross-city prediction performance is significantly improved.

Description

A Cross-City Traffic Prediction Method and System Based on Multimodal Fusion and Spatial Expert Routing Technical Field

[0001] This invention relates to the field of traffic forecasting technology, and more specifically to a cross-city traffic forecasting method and system based on multimodal fusion and spatial expert routing. Background Technology

[0002] Traffic forecasting is a core task of intelligent transportation systems and urban computing, aiming to predict future traffic conditions based on historical traffic data (such as flow rate and speed) to support travel planning, signal control, and road network optimization. In recent years, deep learning methods (such as GCN+RNN) have significantly improved prediction accuracy in single-city scenarios by jointly modeling spatial dependencies and temporal dynamics.

[0003] However, existing models generally rely on sensor data from specific cities, making cross-city migration difficult. To reduce deployment costs, spatiotemporal prediction foundational models (such as OpenCity and UniST) have begun exploring learning general representations from multi-city data to achieve zero-sample predictions for unseen cities. However, these models are still limited to single-modal time series or grid data, ignoring the rich multimodal urban contextual information in real transportation systems (such as satellite imagery, POIs, GIS, etc.). This information can reveal the commonalities in traffic patterns caused by regional functional differences (such as commercial areas, residential areas, and highway intersections), which is crucial for improving cross-city generalization capabilities.

[0004] Currently, building a foundational model for cross-city multimodal spatiotemporal prediction faces two core challenges: First, the availability and quality of multimodal data are highly inconsistent: the types, completeness, and reliability of available modalities vary significantly across different cities. Second, existing multimodal fusion methods typically assume that all modalities are stable and available, lacking robust mechanisms to handle missing modalities and noise interference. Common strategies such as "training independent models for each modality combination" lead to a surge in maintenance costs, while "using only the minimum common modality set" wastes highly informative modalities. Furthermore, fixed fusion weights are difficult to dynamically adjust based on modality quality and task relevance, making them susceptible to interference from low-quality modalities.

[0005] Spatial dependency modeling exhibits strong regional specificity but weak generalization ability: spatial patterns of traffic flow are highly dependent on local urban functions and road network structures (e.g., highways vs. commercial districts). Existing models (such as fixed graph convolution) learn spatial relationships that are strongly coupled with the sensor layout of the training city, making it difficult to transfer to new cities with vastly different structures, severely limiting zero-sample generalization ability.

[0006] In summary, there is an urgent need for a unified architecture that can adaptively filter and fuse high-quality multimodal information while decoupling regional specificity from general spatial patterns, in order to overcome the performance bottleneck of cross-city multimodal traffic prediction basic models. Summary of the Invention

[0007] In view of the above problems, this invention is proposed to provide a cross-city traffic prediction method and system based on multimodal fusion and spatial expert routing to overcome or at least partially solve the above problems. Through two core technologies, adaptive modal filtering and multimodal-driven spatial expert routing, the fundamental problems of "unreliable modalities" and "difficulty in transferring spatial patterns" in cross-city traffic prediction are solved simultaneously. While ensuring robustness and flexibility, it significantly improves zero-sample generalization performance and prediction accuracy, providing key technical support for building a practical and scalable basic model for spatiotemporal traffic prediction.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, embodiments of the present invention provide a cross-city traffic prediction method based on multimodal fusion and spatial expert routing, comprising: S1, acquiring multimodal input data of the target city; S2, encoding each available modality of the multimodal input data to obtain an initial embedding representation of each modality, and projecting all initial embedding representations of modalities onto a unified embedding space to form a multimodal embedding matrix; S3, constructing a modal state matrix, and based on the modal state matrix, performing masking processing on the multimodal embedding matrix to mask unavailable modalities; S4, for each available modality after masking processing, calculating its uncertainty score and task relevance score, and calculating the signal-to-noise ratio (SNR) of the modality based on these two scores; and calculating the signal-to-noise ratio (SNR) of the modality based on the SNR. In step S5, a group of spatial experts is activated based on the multimodal fusion embedding. These spatial experts include modality sharing experts and routing experts. For each activated spatial expert, for each sensor node, its neighboring nodes are selected based on a preset adjacency relationship. The temporal embeddings of the target node and its neighboring nodes are aggregated using a cross-attention mechanism to generate a spatial perception patch embedding. In step S6, the spatial perception patch embeddings of all activated spatial experts are weighted and fused to obtain a fused spatial perception patch embedding. In step S7, the fused spatial perception patch embedding is input to a non-autoregressive spatiotemporal decoder to generate a prediction result for future traffic conditions.

[0009] Preferably, the multimodal input data includes historical traffic status time series mode, as well as one or more of satellite image mode, human activity data mode, and location information mode.

[0010] Preferably, step S2 includes: for satellite image modalities, using a pre-trained ResNet50 model to extract features from the geographic area image corresponding to each traffic sensor node in the target city dataset, to obtain image feature representations. ,in, This represents the number of sensor nodes in the target city dataset. The image feature dimension is used as the initial embedding representation for this modality. For the human activity data modality, a pre-trained BERT model is used to extract features from the surrounding POI category data of all traffic sensor nodes in the target city dataset to obtain the text feature representation. ,in, The text feature dimension is used as the initial embedding representation for this modality; for the location information modality, the latitude and longitude coordinates of each traffic sensor node in the target city dataset are normalized to the interval [-1,1] to form a two-dimensional location feature representation. The initial embedding representation of this modality is used. For historical traffic state time series modalities, the input time series is first normalized by instance, and then divided into non-overlapping patches of length L in the time dimension. Each patch is mapped to a D-dimensional vector through a linear projection layer, and a learnable temporal embedding of the corresponding timestamp is superimposed on it. The resulting patch embedding sequence is concatenated with a learnable CLS token and then input into a multi-head self-attention module incorporating rotational position encoding (RoPE) to obtain the complete temporal modality representation. Extract the CLS tokens from the complete temporal modal representations of all samples in the current batch and average them by batch to obtain the global temporal modal embeddings of traffic sensor nodes in the target city dataset. This serves as the initial embedding representation for the modality. For each modality's initial embedding representation, it is mapped to a unified D-dimensional embedding space through a modality-specific linear layer. A learnable modality embedding storing modal characteristics is initialized for each modality, and this embedding is added to the mapped D-dimensional embeddings of each modality to obtain the corresponding satellite image modalities. Location information modality Human activity data modalities Historical traffic status time series mode ; Satellite image modal Location information modality Human activity data modalities Historical traffic status time series mode The data is concatenated along the modal dimension to form a multimodal embedding matrix. .

[0011] Preferably, in step S4, the signal-to-noise ratio calculation formula is:

[0012]

[0013]

[0014] in, For signal-to-noise ratio, For uncertain fractions, The task relevance score. The available modes after masking. and These are, respectively, a multilayer perceptron with a sigmoid activation function; and a gating function. Defined as:

[0015]

[0016] in, This indicates an indicator function that returns 1 if the condition is true and 0 if the condition is false. () represents the Gumbel-Sigmoid function. In the Gumbel-Sigmoid function... and Let each represent a random variable sampled from a standard Gumbel distribution. This represents the Sigmoid function. This represents a learnable temperature parameter. This represents the hyperparameter that determines the gating threshold.

[0017] Preferably, in step S4, a contrastive learning objective is introduced. This objective is based on an original multimodal embedding and constructs two types of negative samples: the first type is modal-noisy samples, generated by replacing the embedding of a randomly selected available modality from the multimodal embedding matrix with Gaussian noise of the same dimension. The second method is modal dropout samples. This is achieved by updating the position in the modal state matrix corresponding to the random modality to the missing state, and then performing corresponding masking on the multimodal embedding matrix to generate the modal dropout embedding matrix. Using a learnable global token, respectively, and the multimodal embedding matrix... The modal noise embedding matrix and the modal discard embedding matrix Perform cross-attention operation to obtain the corresponding three fused embeddings. , This represents the multimodal fusion representation of the target. and There are two types of negative sample representations, among which Represents the multimodal fusion characterization of noise. This represents the multimodal fusion representation after mode discarding; wherein the differentiable gating mechanism is trained to effectively suppress noisy or discarded modal channels, thereby expecting... and They are close to each other in the embedded space, and far apart from each other. .

[0018] Preferably, S5 includes: for each activated spatial expert and each traffic sensor node Based on the preset adjacency matrix, the one with the highest weight is selected. A neighborhood is formed by several neighboring nodes; based on complete temporal modality representation. The temporal patch embeddings of selected neighbor nodes are weighted according to their corresponding adjacency weights to form a weighted neighbor representation; this weighted neighbor representation is used as the key and value of the cross-attention mechanism, with sensor nodes... The temporal patch embedding itself is used as a query to calculate the cross-attention output; the cross-attention output is then passed through a feedforward network with residual connections and layer normalization to generate the spatial awareness patch embedding corresponding to the spatial expert.

[0019] Preferably, to alleviate the load imbalance problem among multiple routing experts, a load balancing loss function is introduced:

[0020]

[0021] in, To help space experts balance the losses, and These represent the sensor allocation distribution and the routing probability distribution, respectively. For the number of routing experts, router modules Composed of a multilayer perceptron with a softmax activation function This indicates an indicator function that returns 1 if the condition is true and 0 if the condition is false. This represents the number of traffic sensor nodes in the target city dataset.

[0022] Preferably, the total loss function is:

[0023]

[0024] in, For mean square error loss, For modal selector contrast loss, To help space experts balance the losses, and For adjustable weight parameters, To predict the output, Represents the ReLU function. For Euclidean distance, This is a marginal parameter.

[0025] Secondly, embodiments of the present invention provide a cross-city traffic prediction system based on multimodal fusion and spatial expert routing, comprising: a data acquisition module for acquiring multimodal input data of a target city; a multimodal encoding module for encoding each available modality of the multimodal input data to obtain an initial embedding representation of each modality, and projecting the initial embedding representations of all modalities onto a unified embedding space to form a multimodal embedding matrix; an alignment module for constructing a modal state matrix, and masking the multimodal embedding matrix based on the modal state matrix to block unavailable modalities; and an adaptive modal fusion module for calculating the uncertainty score and task relevance score of each available modality after masking, and calculating the signal-to-noise ratio of the modality based on the two; and, based on the signal-to-noise ratio, merging the modality through a differentiable gate. The system employs a control mechanism to dynamically retain high signal-to-noise ratio (SNR) modes and suppress low SNR modes, generating a multimodal fusion embedding. A multimodal-guided spatial expert routing module activates a group of spatial experts based on the multimodal fusion embedding. These spatial experts include mode-sharing experts and routing experts. For each activated spatial expert, neighboring nodes are selected based on a preset adjacency relationship for each sensor node. A cross-attention mechanism is used to aggregate the temporal embeddings of the target node and its neighboring nodes, generating a spatial perception patch embedding. A weighted fusion module performs weighted fusion on the spatial perception patch embeddings of all activated spatial experts to obtain a fused spatial perception patch embedding. A spatiotemporal decoding module inputs the fused spatial perception patch embeddings into a non-autoregressive spatiotemporal decoder to generate a prediction of future traffic conditions.

[0026] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a cross-city traffic prediction method and system based on multimodal fusion and spatial expert routing, which has the following effects: 1. It achieves adaptive and robust fusion for missing multimodal modes and noise, significantly improving model reliability. By introducing a mode selection mechanism based on signal-to-noise ratio (SNR), the present invention can dynamically evaluate the information quality of each available mode: high SNR modes are retained for fusion, and low SNR modes (such as channels affected by noise or information redundancy) are automatically suppressed. Combined with a contrastive learning strategy (constructing mode noise and discarding negative samples), the ability of the mode selector to discriminate low-quality inputs is further strengthened. This mechanism enables the model to stably output high-quality joint representations even when facing real-world scenarios with incomplete modes and varying data quality in different cities, effectively avoiding the performance degradation caused by the introduction of noisy modes in traditional fixed fusion strategies.

[0027] 2. Supports flexible inference for arbitrary modal combinations, reducing deployment and maintenance costs. This invention adopts an integration strategy of "shielding unavailable modalities and utilizing only available modalities," eliminating the need to train multiple dedicated models for different city modalities or rely on a "minimum common modality set" shared by all cities. A single unified architecture can accommodate any input combination from single-modality to full-modality, fully utilizing the unique high-value auxiliary information of each city while significantly reducing computational overhead and system complexity, providing an efficient and feasible solution for large-scale cross-city deployment.

[0028] 3. Overcoming the geographical limitations of spatial modeling and achieving zero-shot generalization across cities: Through a multimodal guided spatial expert routing mechanism, this invention explicitly associates spatial dependency modeling with the semantics of local urban functions (such as commercial areas, highways, and residential areas). Regions with similar multimodal backgrounds (even if located in different cities) will activate the same or similar spatial experts, thereby learning transferable, context-aware spatial interaction patterns. Each expert focuses on modeling the sensor neighborhood relationships of a specific type of region, avoiding the "average" modeling of heterogeneous spatial structures by a single network, and significantly improving the model's zero-shot prediction capability in unseen cities and unknown road network structures.

[0029] 4. An end-to-end collaborative optimization spatiotemporal prediction framework is constructed to improve overall prediction accuracy. The robust joint representation output by the multimodal enhancement module is directly used to guide expert activation and neighbor aggregation in the spatiotemporal prediction module; while the spatially aware representation is jointly decoded with temporal embedding to form a closed-loop optimization. This deep coupling design of multimodality, space, and time enables the model to not only capture long-term temporal dynamics but also accurately characterize local spatial dependencies. Ultimately, it achieves prediction accuracy superior to existing single-modal base models and traditional multimodal methods on multiple real cross-city traffic datasets. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0031] Figure 1 is a schematic diagram of the cross-city traffic prediction method based on multimodal fusion and spatial expert routing provided in an embodiment of the present invention; Figure 2 is a block diagram of the cross-city traffic prediction system based on multimodal fusion and spatial expert routing provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention discloses a cross-city traffic prediction method based on multimodal fusion and spatial expert routing, as shown in Figure 1. The method includes: S1, acquiring multimodal input data for the target city; S2, encoding each available modality of the multimodal input data to obtain an initial embedding representation for each modality, and projecting all initial embedding representations of the modalities onto a unified embedding space to form a multimodal embedding matrix; S3, constructing a modal state matrix, and masking the multimodal embedding matrix based on the modal state matrix to block unavailable modalities; S4, calculating the uncertainty score and task relevance score for each available modality after masking, and calculating the signal-to-noise ratio (SNR) of that modality based on these two scores; and using a differentiable gate based on the SNR. The control mechanism dynamically retains high signal-to-noise ratio modes and suppresses low signal-to-noise ratio modes to generate multimodal fusion embeddings; S5, a group of spatial experts are activated based on the multimodal fusion embeddings; among them, spatial experts include mode-sharing experts and routing experts; for each activated spatial expert, for each sensor node, its neighboring nodes are selected based on a preset adjacency relationship, and the temporal embeddings of the target node and its neighboring nodes are aggregated using a cross-attention mechanism to generate spatial perception patch embeddings; S6, the spatial perception patch embeddings of all activated spatial experts are weighted and fused to obtain fused spatial perception patch embeddings; S7, the fused spatial perception patch embeddings are input into a non-autoregressive spatiotemporal decoder to generate prediction results for future traffic conditions.

[0034] To address the inconsistency in modal availability across different cities, this invention employs a strategy of shielding unavailable modalities and directly utilizing available modalities. This method improves generalization ability while reducing computational overhead. The overall architecture is designed to support this flexible and efficient multimodal integration. In modeling the impact of modal quality and modal correlation, the invention is inspired by Shannon's second law, which states that channels with high signal-to-noise ratio (SNR) enable reliable information transmission. Therefore, low SNR modalities typically carry limited noise information, reducing their value for downstream tasks. After encoding the representation of each available modality, its SNR is estimated, and a modality selector is applied to disable channels with insufficient SNR. The remaining high-quality modalities are then fused to form a robust joint representation. This invention introduces a contrastive learning strategy to guide the selector to suppress low SNR inputs, thereby ensuring robust and adaptive multimodal integration.

[0035] To address the challenge of strong region-specificity but weak generalization in spatially dependent modeling, this invention utilizes multimodal data to guide the learning of spatial patterns conditioned on the current context. The spatial structures of different cities vary significantly, and using a single neural network to uniformly model all regions leads to suboptimal results. Regions with similar multimodal backgrounds, even those belonging to different cities, often exhibit comparable spatial structures and traffic patterns. Leveraging this characteristic, specialized spatial experts are activated using available multimodal information, each designed to capture distinct spatiotemporal patterns. For each spatial expert, the neighbors of the target sensor are identified, and the correlation between the target and its neighboring sensors is modeled. This enables the construction of spatially perceptual representations. Finally, temporal embeddings are jointly decoded with these representations to generate the final traffic prediction.

[0036] The implementation process of this invention is described in detail below: The multimodal enhancement module is used to model and select available modalities, thereby improving the reliability of input information. In order to effectively utilize multimodal information, a modality-specific encoder is used to extract features from each modality.

[0037] For satellite image modalities, a pre-trained ResNet50 model is used for feature extraction to obtain image feature representations. ,in, This represents the number of traffic sensor nodes in the target city dataset. Represents the dimension of image features.

[0038] For human activity data modalities, a pre-trained BERT model is used to extract features from the POI category data surrounding the traffic sensor nodes, resulting in text feature representations. ,in Represents the dimension of text features.

[0039] For location information modes, the latitude and longitude coordinates are normalized to The interval is used to obtain a two-dimensional location feature representation. .

[0040] For historical traffic state time series modes ,in Representing the time dimension, firstly, the time series modes of historical traffic conditions... Instance normalization is performed, and then the instances are divided into non-overlapping patches along the time dimension to obtain the partitioned three-dimensional time series. ,in This represents the length of each patch. Further, each patch is projected through a linear layer (MLP) onto... The dimensional embedding space is then used, along with the learnable temporal embeddings corresponding to the timestamps, to obtain... These embeddings are concatenated with an additional learnable token CLS to obtain the first encoded features. Based on the first coding feature By incorporating a multi-head attention mechanism with rotational position encoding (RoPE), a complete temporal modality representation is obtained. Extract the complete temporal modal representation of all samples in the current batch. The global temporal modal embedding of each traffic sensor node in the target city dataset is obtained by averaging the CLS tokens in the dataset over batches. .

[0041] Because different pre-trained encoders extract features with inconsistent dimensions, a modality-specific linear layer is used to project the features of each modality onto a unified surface. The satellite image modality is obtained by constructing a 3D embedding space and combining it with learnable corresponding modality embeddings to enhance semantic discriminability. Location information modality Human activity data modalities Historical traffic status time series mode Satellite image modalities Location information modality Human activity data modalities Historical traffic status time series mode The components are spliced ​​together to form a unified multimodal embedding matrix. : .

[0042] To handle the case of missing modes, a modal state matrix is ​​defined. The elements represent the availability of the corresponding mode for each sensor (1 for available, 0 for missing). This is based on the modal state matrix. Apply a binary mask to the multimodal embedding matrix to exclude the embedding of missing modes.

[0043] In the mode selector, reliable modes with high signal-to-noise ratio (SNR) are evaluated and filtered. Each mode is treated as an independent information transmission channel, with its information transmission channel capacity... Positively correlated with signal-to-noise ratio (SNR): Modes with excessively low signal-to-noise ratios have their information overwhelmed by noise and should be filtered out.

[0044] For each available mode That is, from the multimodal embedding matrix The corresponding modality embeddings extracted are used to estimate their uncertainty fractions using a multilayer perceptron (MLP) in conjunction with a sigmoid activation function. Task relevance score :

[0045]

[0046] Based on uncertainty score Task relevance score Calculate the signal-to-noise ratio for each mode:

[0047] Based on the signal-to-noise ratio (SNR) of each mode, low SNR modes are gated in a differentiable manner, using a Gumbel-Sigmoid mechanism in conjunction with a pass-through estimator (STE) to approximate the binary mask. The gating function... Defined as:

[0048]

[0049] in, This indicates an indicator function that returns 1 if the condition is true, and 0 if the condition is false. This refers to the Gumbel-Sigmoid function. In the Gumbel-Sigmoid function... and Let each represent a random variable sampled from a standard Gumbel distribution. This represents a learnable temperature parameter. This represents the hyperparameter that determines the gating threshold. This represents the Sigmoid function. This represents the maximum signal-to-noise ratio among the available modes. The OR operator. To ensure that at least the most effective information is retained, the mode with the highest signal-to-noise ratio is always retained.

[0050] To enhance the discriminative ability of the selector, this invention introduces a contrastive learning objective. For a given objective, two types of negative samples are generated. The first type of negative sample is a modality-noisy sample, which replaces the embedding representation of a randomly selected available modality in the multimodal embedding matrix with Gaussian noise, thereby generating a modality-noisy embedding matrix. The second type of negative sample is the modality dropout sample, which means that the position in the modality-state matrix corresponding to the random modality is updated to the missing state, that is, the modality state is updated to be missing, in order to generate the modality dropout embedding matrix. Through a learnable global token and a multimodal embedding matrix. Modal noise embedding matrix and modal discarding embedding matrix Cross-attention is performed to obtain fused embeddings. , This represents the multimodal fusion representation of the target. and There are two types of negative sample representations, among which Represents the multimodal fusion characterization of noise. This represents the multimodal fusion characterization after mode discarding. The purpose of the mode selector is to eliminate noisy mode channels, hence the expectation is... and Approaching within the embedded space, and with the target Distance. Contrast loss is defined as:

[0051] in, Represents the ReLU function. For Euclidean distance, This is a marginal parameter. This parameter defines a minimum separation distance: when the Euclidean distance between the noisy or discarded representation and the target representation is greater than a certain value... When this happens, the corresponding penalty term will be invalidated (by setting it to zero using the ReLU function).

[0052] To fully utilize multimodal information to capture local spatial dependencies while maintaining generalization ability, a multimodal spatiotemporal prediction module is proposed, which includes a multimodal guided spatial aggregation block and a spatiotemporal decoder.

[0053] Multimodal guided spatial aggregation blocks aim to leverage multimodal information to activate context-specific spatial experts to capture local spatial dependencies. Specifically, the spatial experts are designed to interact with each sensor and its associated spatial dependencies. The interaction between nearest neighbor sensors is modeled to efficiently capture local spatial dependencies.

[0054] For each space expert, for each traffic sensor According to the adjacency matrix Select the one with the highest weight The neighborhood is composed of neighboring nodes. Based on complete temporal modal representation Embed the timing patches of these neighboring nodes According to its corresponding adjacency weight The neighbor representations are reweighted to form a weighted representation. Based on these weighted neighbor representations, the key and value of the cross-attention mechanism are constructed. The cross-attention mechanism is then applied to each sensor i:

[0055] in, For sensors Embedding of timing coding patches.

[0056] Next, output the cross-attention. Spatial-aware patch embeddings are generated using a feedforward network with residual connections and layer normalization. .

[0057]

[0058] Based on the above construction method of space experts, two types of space experts are constructed: Individual Modality Sharing Expert (SE-M) and There are 1 routing expert (SE-i). The number of modal shared experts is... Equals the number of input modes, for each mode activated by the mode selector. The corresponding SE-M is activated, meaning that each activated mode corresponds to the activation of a mode sharing expert.

[0059] For the routing expert (SE-i), multimodal fusion embedding is used. As a router module The router calculates the routing probability based on the input and activates the routing expert (SE-i) with the highest probability.

[0060] To alleviate load imbalance among routing experts, a balancing loss is introduced. This encourages a uniform distribution of sensors among all experts in each training batch:

[0061]

[0062] in, and Representing the sensor allocation distribution and routing probability distribution respectively, the router module... Composed of a multilayer perceptron with a softmax activation function This indicates an indicator function that returns 1 if the condition is true and 0 if the condition is false.

[0063] Finally, using the probabilities obtained from the selector and router, the outputs of the modality sharing expert and the routing expert are weighted and summed to generate a fused spatially aware patch embedding. .

[0064] To capture spatial correlations during time-series decoding and adapt to different output lengths, we employed a non-autoregressive spatiotemporal decoder. The last token output by the spatiotemporal encoder... (Contains historical information) Copy Next, construct the decoder input. ,in To predict the sequence length. Embedded with spatial awareness patch Common input Transformer decoder: .

[0065] Finally, a linear layer is used to generate the predicted values, and inverse instance normalization is applied to obtain the final predicted output. .

[0066] Furthermore, this invention performs optimization training based on the total loss function.

[0067] Specifically, the total loss function is a weighted average of the prediction loss, the contrast loss, and the balancing loss:

[0068] in For mean square error loss, For modal selector contrast loss, To help space experts balance the losses, and These are adjustable weight parameters.

[0069] By jointly optimizing the above total loss function The system is trained to adaptively utilize and filter multimodal information and effectively capture local spatiotemporal dependencies, thereby achieving powerful cross-city zero-sample traffic prediction capabilities.

[0070] This method was tested in traffic condition prediction applications. Experiments were conducted on five large-scale traffic datasets: Taxi-NYC, Taxi-CHI, and the SD, GBA, and GLA datasets from LargeST. Dataset statistics are shown in Table 1. Satellite imagery data was acquired using Google Earth Engine, while human activity text data (specifically, POI category information) was collected from the Foursquare platform. For each sensor, a 1 km × 1 km area was cropped centered on it to obtain the corresponding satellite imagery; for human activity text data, all POI category information within the same area was collected. Each dataset was divided into training, validation, and test sets in a 7:1:2 ratio, ordered chronologically.

[0071] Table 1: Dataset Statistics

[0072] The MoST method of this invention is compared with state-of-the-art models in the field of spatiotemporal prediction, including Historical Average (HA), Vector Autoregression (VAR), STGCN, GWN, ASTGCN, STWA, BigST, PatchSTG, and the basic model OpenCity. The evaluation metrics used are Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). During training, data from the previous 10 days are used to predict the next 3 days. The overall performance is shown in Table 2.

[0073] Table 2: Performance comparison of MOST in zero-sample setting and baseline model in full-sample and zero-sample settings

[0074] MoST outperforms most baseline models in prediction error across all datasets in the zero-shot setting, demonstrating excellent cross-city generalization ability. Specifically, we can observe the following: the heuristic method HA produces the weakest performance, mainly because it relies on historical averages and fails to capture the dynamic temporal and spatial variations present in real-world traffic data. In the full scenario, the end-to-end spatiotemporal models STGCN, GWN, and ASTGCN exhibit strong predictive capabilities. However, their generalization performance on the SD dataset is poor, and their robustness is limited. In contrast, BigST and PatchSTG, designed specifically for large-scale datasets, effectively model spatial patterns and perform well on the SD dataset.

[0075] However, these end-to-end baselines are constrained by fixed topologies, limiting their adaptability to unseen datasets with completely different spatial configurations.

[0076] While both STGCN and GWN perform strongly in full-sample settings, their performance drops significantly in zero-sample settings. This degradation occurs because these models rely on predefined static graph structures and cannot adapt to changing spatiotemporal dependencies, thus limiting their generalization ability across cities. In contrast, the base model OpenCity demonstrates stronger generalization ability in zero-sample settings because it only needs to be trained once and can be applied to different cities without retraining. Although OpenCity performs competitively in zero-sample settings and outperforms several end-to-end baselines, it still lags behind the proposed model MoST across all datasets. Notably, all these baselines rely on a single modality, limiting their ability to capture the rich contextual information embedded in multimodal data, which ultimately hinders their predictive accuracy.

[0077] This invention provides a cross-city traffic prediction system based on multimodal fusion and spatial expert routing, as shown in Figure 2. It includes: a data acquisition module for acquiring multimodal input data from the target city; a multimodal encoding module for encoding each available modality of the multimodal input data to obtain an initial embedding representation for each modality, and projecting all initial embedding representations of the modalities onto a unified embedding space to form a multimodal embedding matrix; an alignment module for constructing a modal state matrix and, based on the modal state matrix, masking the multimodal embedding matrix to remove unavailable modalities; and an adaptive modal fusion module for calculating the uncertainty score and task relevance score for each available modality after masking, and calculating the signal-to-noise ratio (SNR) of that modality based on these two scores; and, according to the SNR, passing the data through a differentiable gate... The system employs a control mechanism to dynamically retain high signal-to-noise ratio (SNR) modes and suppress low SNR modes, generating a multimodal fusion embedding. A multimodal-guided spatial expert routing module activates a group of spatial experts based on the multimodal fusion embedding. These spatial experts include mode-sharing experts and routing experts. For each activated spatial expert, neighboring nodes are selected based on a preset adjacency relationship for each sensor node. A cross-attention mechanism is used to aggregate the temporal embeddings of the target node and its neighboring nodes, generating a spatial perception patch embedding. A weighted fusion module performs weighted fusion on the spatial perception patch embeddings of all activated spatial experts to obtain a fused spatial perception patch embedding. A spatiotemporal decoding module inputs the fused spatial perception patch embeddings into a non-autoregressive spatiotemporal decoder to generate a prediction of future traffic conditions.

[0078] Based on the same inventive concept, and since the principle by which the system solves the problem is similar to that of the aforementioned method, the implementation of the aforementioned method can be referred to, and the repeated parts will not be described again.

[0079] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0080] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-city traffic prediction method based on multimodal fusion and spatial expert routing, characterized in that, include: S1. Acquire multimodal input data for the target city; S2. Encode each available modality of the multimodal input data to obtain an initial embedding representation for each modality, and project all initial embedding representations of the modalities onto a unified embedding space to form a multimodal embedding matrix; S3. Construct a modality state matrix, and based on the modality state matrix, perform masking on the multimodal embedding matrix to mask unavailable modalities; S4. For each available modality after masking, calculate its uncertainty score and task relevance score, and calculate the signal-to-noise ratio (SNR) of that modality based on these two scores; Based on the SNR, pass the signal through a differentiable gate... The control mechanism dynamically retains high signal-to-noise ratio modes and suppresses low signal-to-noise ratio modes to generate multimodal fusion embeddings; S5, a group of spatial experts are activated based on the multimodal fusion embeddings; among them, spatial experts include mode-sharing experts and routing experts; for each activated spatial expert, for each sensor node, its neighboring nodes are selected based on a preset adjacency relationship, and the temporal embeddings of the target node and its neighboring nodes are aggregated using a cross-attention mechanism to generate spatial awareness patch embeddings; S6, all activated spatial experts are weighted and fused in the spatial awareness patch embeddings to obtain fused spatial awareness patch embeddings; S7 embeds the fused spatial perception patch into the input to the non-autoregressive spatiotemporal decoder to generate predictions of future traffic conditions.

2. The method as described in claim 1, characterized in that, Multimodal input data includes historical traffic status time series modalities, as well as one or more of satellite imagery modalities, human activity data modalities, and location information modalities.

3. The method as described in claim 2, characterized in that, S2 includes: for satellite image modalities, using a pre-trained ResNet50 model to extract features from the geographic area image corresponding to each traffic sensor node in the target city dataset, to obtain image feature representations. ,in, This represents the number of traffic sensor nodes in the target city dataset. The image feature dimension is used as the initial embedding representation for this modality. For the human activity data modality, a pre-trained BERT model is used to extract features from the surrounding POI category data of all traffic sensor nodes in the target city dataset to obtain the text feature representation. ,in, The text feature dimension is used as the initial embedding representation for this modality; for the location information modality, the latitude and longitude coordinates of the traffic sensor nodes in the target city dataset are normalized to the interval [-1,1] to form a two-dimensional location feature representation. The initial embedding representation of this modality is used. For historical traffic state time series modalities, the input time series is first normalized by instance, and then divided into non-overlapping patches of length L in the time dimension. Each patch is mapped to a D-dimensional vector through a linear projection layer, and a learnable temporal embedding of the corresponding timestamp is superimposed on it. The resulting patch embedding sequence is concatenated with a learnable CLS token and then input into a multi-head self-attention module incorporating rotational position encoding (RoPE) to obtain the complete temporal modality representation. Extract the CLS tokens from the complete temporal modal representations of all samples in the current batch and average them by batch to obtain the global temporal modal embeddings of traffic sensor nodes in the target city dataset. This serves as the initial embedding representation for the modality. For each modality's initial embedding representation, it is mapped to a unified D-dimensional embedding space through a modality-specific linear layer. A learnable modality embedding storing modal characteristics is initialized for each modality, and this embedding is added to the mapped D-dimensional embeddings of each modality to obtain the corresponding satellite image modalities. Location information modality Human activity data modalities Historical traffic status time series mode ; Satellite image modal Location information modality Human activity data modalities Historical traffic status time series mode The data is concatenated along the modal dimension to form a multimodal embedding matrix. 。 4. The method as described in claim 1, characterized in that, In S4, the signal-to-noise ratio calculation formula is: in, For signal-to-noise ratio, For uncertain fractions, The task relevance score. The available modes after masking. and These are, respectively, a multilayer perceptron with a sigmoid activation function; and a gating function. Defined as: in, This indicates an indicator function that returns 1 if the condition is true and 0 if the condition is false. () represents the Gumbel-Sigmoid function, in the Gumbel-Sigmoid function and Let each represent a random variable sampled from a standard Gumbel distribution. This represents a learnable temperature parameter. This represents the hyperparameter that determines the gating threshold. This represents the Sigmoid function. This represents the maximum signal-to-noise ratio among the available modes. The OR operator.

5. The method as described in claim 4, characterized in that, In step S4, a contrastive learning objective is introduced. Based on an original multimodal embedding, this objective constructs two types of negative samples: the first is a modality-noisy sample, generated by replacing the embedding of a randomly selected available modality from the multimodal embedding matrix with Gaussian noise of the same dimension. The second method is modal dropout samples. This is achieved by updating the position in the modal state matrix corresponding to the random modality to the missing state, and then performing corresponding masking on the multimodal embedding matrix to generate the modal dropout embedding matrix. Using a learnable global token, respectively, and the multimodal embedding matrix... The modal noise embedding matrix and the modal discard embedding matrix Perform cross-attention operation to obtain the corresponding three fused embeddings. ， This represents the multimodal fusion representation of the target. and There are two types of negative sample representations, among which Represents the multimodal fusion characterization of noise. This represents the multimodal fusion representation after mode discarding; wherein the differentiable gating mechanism is trained to effectively suppress noisy or discarded modal channels, thereby expecting... and They are close to each other in the embedded space, and far apart from each other. 。 6. The method as described in claim 5, characterized in that, S5 includes: for each activated spatial expert and each traffic sensor node Based on the preset adjacency matrix, the one with the highest weight is selected. A neighborhood is formed by several neighboring nodes; based on complete temporal modality representation. The temporal patch embeddings of selected neighbor nodes are weighted according to their corresponding adjacency weights to form a weighted neighbor representation; the weighted neighbor representation is used as the key and value of the cross-attention mechanism, with sensor nodes... The temporal patch embedding itself is used as a query to calculate the cross-attention output; the cross-attention output is then passed through a feedforward network with residual connections and layer normalization to generate the spatial awareness patch embedding corresponding to the spatial expert.

7. The method as described in claim 6, characterized in that, To alleviate the load imbalance problem among multiple routing experts, a load balancing loss function is introduced: in, To help space experts balance the losses, and These represent the sensor allocation distribution and the routing probability distribution, respectively. For the number of routing experts, router modules It consists of a multilayer perceptron with a softmax activation function. This indicates an indicator function that returns 1 if the condition is true and 0 if the condition is false. This represents the number of traffic sensor nodes in the target city dataset.

8. The method as described in claim 7, characterized in that, The total loss function is: in, For mean square error loss, For modal selector contrast loss, To help space experts balance the losses, and For adjustable weight parameters, To predict the output, Represents the ReLU function. For Euclidean distance, This is a marginal parameter.

9. A cross-city traffic prediction system based on multimodal fusion and spatial expert routing, used to implement the method described in any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire multimodal input data from the target city. The multimodal coding module is used to encode each available modality of the multimodal input data separately to obtain the initial embedding representation of each modality, and project the initial embedding representations of all modalities onto a unified embedding space to form a multimodal embedding matrix; Alignment module: Used to construct the modal state matrix and, based on the modal state matrix, to mask the multimodal embedding matrix, blocking out unusable modes; An adaptive modality fusion module calculates the uncertainty score and task relevance score for each available modality after masking, and calculates the signal-to-noise ratio (SNR) of that modality based on these scores. Based on the SNR, a differentiable gating mechanism dynamically retains high SNR modes and suppresses low SNR modes, generating a multimodal fusion embedding. A multimodal guided spatial expert routing module activates a group of spatial experts based on the multimodal fusion embedding. These spatial experts include modality-sharing experts and routing experts. For each activated spatial expert, neighboring nodes are selected based on a preset adjacency relationship for each sensor node. A cross-attention mechanism is used to aggregate the temporal embeddings of the target node and its neighboring nodes to generate a spatial perception patch embedding. A weighted fusion module performs weighted fusion of the spatial perception patch embeddings of all activated spatial experts to obtain a fused spatial perception patch embedding. A spatiotemporal decoding module inputs the fused spatial perception patch embeddings into a non-autoregressive spatiotemporal decoder to generate a prediction of future traffic conditions.

Images