High-speed rail OD passenger flow prediction method based on large language model

Abstract

The invention discloses a high-speed rail OD passenger flow prediction method based on a large language model. The method comprises the following steps: acquiring a historical OD sequence of a plurality of time steps for a target station; based on the historical OD sequence, an OD passenger flow prediction result of a future time step is obtained by using a trained large language model, and the large language model comprises an input data processing module, a spatial-temporal feature extraction module, a sparse attention large model module and a Gaussian mixture prediction module. The input data processing module is used for dividing the historical OD sequence into time components of different modes, the spatial-temporal feature extraction module is used for capturing spatial and temporal dependence of the historical OD sequence, and the sparse attention large model module is used for obtaining attention output based on a sparsity prior matrix. The Gaussian mixture prediction module is used for adopting a limited number of components to approximate multi-modal distribution, realizing multi-modal modeling and further outputting probability distribution. According to the invention, the accuracy and robustness of OD passenger flow prediction are improved.

Description

Technical Field

[0001] This invention relates to the field of passenger flow forecasting technology, and more specifically, to a method for forecasting high-speed rail OD passenger flow based on a large language model. Background Technology

[0002] With the continuous expansion of the high-speed rail network and the dramatic fluctuations in passenger demand over short timescales, achieving dynamic matching of capacity and demand at the operational level has become one of the core challenges facing operations departments. Against this backdrop, short-term origin-destination (OD) passenger flow forecasting has gradually become a key technology for high-speed rail operations management. On the one hand, OD flow can characterize travel connections between different stations, reflecting the spatial distribution of passenger flow with more granular detail compared to simple inbound / outbound flow. On the other hand, accurate OD forecasting can provide direct basis for train operation plans, seat allocation, fare strategies, and emergency dispatching, helping to improve capacity utilization and reduce operating costs.

[0003] In existing technologies, high-speed rail passenger flow forecasting schemes focus on station-level arrival and departure volume prediction, with scarce predictions for origin-destination (OD) pairs and a lack of uncertainty estimation. For example, some schemes propose GC-LSTM (Graph Convolutional Long Short-Term Memory Network) for predicting passenger flow at high-speed rail stations. Other schemes utilize departure and arrival patterns to model the deterministic part and use a Seq2Seq model with time attention, combined with historical fluctuations and weather information, to predict the stochastic part. However, these schemes all stop at station-level passenger flow forecasting and do not provide confidence intervals. Recent research has coupled Ensemble Empirical Mode Decomposition (EEMD) with Multivariate Support Vector Machines (MSVM) and verified its effectiveness on high-speed rail OD data, but it also lacks probability output.

[0004] Analysis reveals the following main shortcomings in current technologies for traffic passenger flow forecasting, high-speed rail passenger flow forecasting, and the application of Large LLM (LLM) models to traffic time-series data:

[0005] (1) Existing methods cannot effectively handle the sparsity and long-tail distribution of high-speed rail OD data, resulting in low prediction accuracy. In the high-speed rail system, the OD matrix is ​​large, but most ODs have extremely low passenger flow or even zero for a long time, exhibiting highly sparse and long-tailed distribution characteristics. This directly leads to low model training efficiency, high learning difficulty, and unstable prediction performance.

[0006] (2) Existing studies generally set a single embedding for each node, assuming that the OD matrix is ​​a "symmetric" or approximately symmetric structure, ignoring the fact that sites are heterogeneous. This modeling defect leads to misestimation of spatial correlation and weakening of directional dependence.

[0007] (3) Existing methods generally lack characterization of uncertainties such as "ticket changes, refunds, and waitlists," and can only provide point predictions. In the high-speed rail passenger flow system, ticket changes, refunds, and waitlists are all high-frequency behaviors. These behaviors can lead to inconsistencies between historical origin-destination (OD) data and actual travel, and point prediction models cannot reflect confidence levels, resulting in increased scheduling risks. However, existing methods generally do not have probability distribution outputs, cannot provide confidence intervals or risk indicators, and cannot support "risk-aware" scheduling in operational decisions. Therefore, the prediction results of existing technologies are unstable, unreliable, and have uncontrollable risks.

[0008] (4) Most existing LLM-based solutions for traffic forecasting suffer from problems such as high cost of full fine-tuning, failure to utilize node role differences, and insufficient matching between structure and OD data. Therefore, existing LLM-based methods in high-speed rail OD forecasting suffer from high computational cost, easy overfitting, and poor generalization performance. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a high-speed rail OD passenger flow prediction method based on a large language model. This method includes the following steps:

[0010] For the target site, obtain historical OD sequences at multiple time steps;

[0011] Based on the historical OD sequence, the OD passenger flow prediction results for future time steps are obtained using a trained large language model.

[0012] The large language model includes an input data processing module, a spatiotemporal feature extraction module, a sparse attention large model module, and a Gaussian mixture prediction module. The input data processing module is used to divide the historical OD sequence into temporal components of different modes. The spatiotemporal feature extraction module is used to capture the spatial and temporal dependencies of the historical OD sequence. The sparse attention large model module is used to obtain attention output based on a sparse prior matrix. The Gaussian mixture prediction module is used to approximate a multimodal distribution with a finite number of components to achieve multimodal modeling and then output a probability distribution.

[0013] Compared with the prior art, the advantages of the present invention are mainly reflected in the following aspects:

[0014] (1) To address the "sparseness and long-tail distribution problem," the model's ability to focus on key OD pairs is enhanced. By designing a sparse prior attention mechanism, the model is guided to allocate more reasonable attention weights, thereby strengthening the modeling of major OD pairs and weakening the impact of ineffective OD pairs. Ultimately, this improves the overall prediction accuracy and training efficiency.

[0015] (2) To address the problem of insufficient directional modeling, a dual-node embedding mechanism for OD is constructed. This invention introduces departure station embedding and arrival station embedding. This mechanism can explicitly distinguish the "departure role" and "arrival role" of OD, thereby capturing the differences between heterogeneous ODs, strengthening the ability to express spatial structure, and enhancing the ability to model OD pairs.

[0016] (3) To address the issue of "uncertainty that cannot be quantified," this invention provides probabilistic predictions that can be used for operational scheduling. This invention employs a Gaussian Mixture Model (GMM) to directly predict the mean and variance of OD (Original Demand) traffic. This approach provides confidence intervals, supporting more robust capacity allocation and emergency scheduling.

[0017] (4) To address the issues of high computational cost and overfitting in LLM, a partially frozen sparse attention LLM is proposed. This invention employs a partial freezing mechanism for large models to reduce training costs. Simultaneously, a sparse attention mechanism is introduced to improve computational efficiency and match the OD structure. This achieves efficient training and adaptability to high-dimensional sparse OD data. Furthermore, this invention proposes a unified prediction framework that simultaneously possesses high-dimensional sparse structure adaptation capability, OD directionality expression capability, uncertainty modeling capability, and efficient LLM enhancement capability, thereby improving the accuracy, stability, and risk perception capability of high-speed rail OD prediction.

[0018] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0020] Figure 1 This is a flowchart of a high-speed rail OD passenger flow prediction method based on a large language model according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a high-speed rail OD passenger flow prediction method based on a large language model according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a residual unit structure according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a large model structure according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram comparing the performance of different models on a high-speed rail dataset according to an embodiment of the present invention.

[0025] Figure 6This is a partial OD pair visualization diagram of the STGM-LLM prediction results in HSR0206 according to an embodiment of the present invention;

[0026] Figure 7 This is a partial OD pair visualization diagram of the STGM-LLM prediction results in HSR0703 according to an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram comparing the performance of different models on a subway dataset according to an embodiment of the present invention;

[0028] Figure 9 This is a partial OD pair visualization diagram of the STGM-LLM prediction results in MetroBJ2016 according to an embodiment of the present invention;

[0029] Figure 10 This is a schematic diagram of model performance under different hyperparameters according to an embodiment of the present invention. Detailed Implementation

[0030] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0032] See Figure 1 As shown, the high-speed rail OD passenger flow prediction method based on a large language model includes: step S1, obtaining historical OD sequences for multiple time steps for the target station; step S2, obtaining OD passenger flow prediction results for future time steps based on the historical OD sequences using a trained large language model.

[0033] In summary, the large language model designed in this invention is a spatial-temporal sparse attention Gaussian mixture large language model (STGM-LLM) suitable for predicting the uncertainty of passenger flow in high-speed rail OD (Origin-Destination) traffic. The model architecture includes an input data processing module, a spatial-temporal feature extraction module, a sparse attention large language model backbone (or sparse attention large model module), a two-node embedding module, a Gaussian mixture prediction module, and training and inference mechanisms. First, a partially parameter-frozen sparse attention GPT-2 focuses on structurally relevant OD interactions, filtering out noise in most empty OD matrices. Second, the spatiotemporal feature extraction module captures spatial and temporal dependencies; for example, this module includes two-node embedding, a ResNet encoder, and a Long Short-Term Memory (LSTM)-based encoder. Third, the Gaussian mixture layer outputs the mean, variance, and weights required for uncertainty prediction. Furthermore, the loss function consists of negative log-likelihood (NLL) and mean absolute error (MAE). NLL ensures reliable uncertainty estimation, while MAE improves point prediction accuracy; the combination of both balances deterministic accuracy and probabilistic calibration.

[0034] I. Problem Definition

[0035] The problem addressed in this invention belongs to the field of high-dimensional spatiotemporal sequence prediction, specifically the uncertainty prediction problem of passenger flow at high-speed rail origin-destination (OD) points. For ease of explanation, the problem is formally defined as follows.

[0036] Definition 1: OD Data Structure Definition. Suppose a high-speed rail network contains N stations. At any time step t, an OD flow matrix can be constructed:

[0037]

[0038] Among them, X t (i,j) represents the passenger flow from station i to station j departing at time t; the OD matrix is ​​a directed matrix, generally having X t (i,j)≠X t (j,i). The historical OD sequence can be represented as:

[0039]

[0040] Definition 2: Prediction Task Definition. Given a historical OD sequence over the most recent T time steps:

[0041]

[0042] The goal is to predict the corresponding OD structure in the next P time steps:

[0043]

[0044] in,

[0045] II. Preliminary Knowledge

[0046] Modern large-scale language models (LLMs) are generally based on the Transformer architecture, which mainly consists of the following core structures:

[0047] Token Representation and Input Embedding: The Transformer accepts sequential input and maps each step (token) in the sequence to a fixed-dimensional vector representation. In natural language processing, tokens are typically words or subwords. In this invention, tokens correspond to spatiotemporal feature fragments of OD traffic.

[0048] Multi-Head Self-Attention: Self-attention is the core computational structure of the Transformer, its function being to calculate the association weights between any two positions in a sequence. This mechanism calculates the associated weights for each value through the dot product between the query and the value. Specifically, given d columns... k The query and key (assuming the query and key have the same number of columns) and the number of columns d v The attention mechanism is defined as follows: (value).

[0049]

[0050] Feed-Forward Network (FFN). Each Transformer Block also includes:

[0051] y = FFN(x) = W2·ReLU(W1x) (6)

[0052] Residual connections and layer normalization (Residual and LayerNorm). LayerNorm is used for stable training. Residual structures enhance gradient flow capability:

[0053] z = x + AttentionBlock(x) (7)

[0054] LSTM (Long Short-Term Memory) model. The core idea of ​​LSTM is to control the memory and forgetting of information through "cell states" and "gating mechanisms." A typical LSTM unit contains four main computational structures: a forgetting gate, an input gate, candidate states, and an output gate. The computation process of LSTM can be described by the following formula:

[0055] The Forget Gate determines how much old information is forgotten:

[0056] f t =σ(W f [h t-1 ,x t ]+b f (8)

[0057] Among them, f t The value of ∈(0,1) controls the memory c from the previous time step. t-1 The percentage that is retained.

[0058] The input gate determines how much new information is received:

[0059] i t =σ(W t [h t-1 ,x t ]+b i (9)

[0060] Candidate cell states generate new potential memories:

[0061] c t =tanh(W c [h t-1 ,x t ]+b c (10)

[0062] Update cell state by combining retained and new information:

[0063] c t =f t ⊙c t-1 +i t ⊙c t (11)

[0064] Here, ⊙ represents element-wise multiplication.

[0065] The output gate determines how much output is in the current hidden state:

[0066] o t =σ(W o [h t-1 x t ]+bo (12)

[0067] Generate the current hidden state as the final output of the LSTM.

[0068] h t =o t ⊙tanh(c t (13)

[0069] III. The Large Language Model Structure Designed in This Invention

[0070] The prediction model based on a large language model proposed in this invention is a sequence modeling framework that combines multi-scale spatiotemporal feature extraction, sparse attention structure, and probabilistic prediction mechanism. The overall structure is as follows: Figure 2 As shown, it mainly includes: a spatiotemporal feature extraction module (including a residual module, an LSTM encoding module, and a two-node embedding module), a sparse attention large model module, a Gaussian mixture regression module, and a loss function, etc.

[0071] Residual Module: To effectively capture heterogeneous temporal dynamics at different scales, in one embodiment, the historical OD sequence is decomposed into three temporal resolutions: weekly trend X w Daily cycle X d and real-time fluctuations X r Each temporal component is processed by ResNet to extract high-level spatiotemporal features. The structure of the residual unit is as follows: Figure 3 As shown.

[0072] Each unit consists of two two-dimensional convolutional layers with a kernel size of 3×3:

[0073] H1 = Conv 3×3 (F in ),H2=Conv 3×3 (H1) (14)

[0074] In the formula The input feature tensor is H1, and the intermediate features are H2. A 3×3 kernel ensures that each OD pair interacts with its local spatial neighborhood while preserving the global OD matrix structure. Leaky ReLU activation is applied between the two convolutional layers.

[0075]

[0076] Here, α is a small negative slope coefficient. Compared to standard ReLU, Leaky ReLU can prevent the problem of "dead neurons" and allows slight negative activation to pass through, which is particularly beneficial when dealing with sparse OD flow and avoiding the loss of weak signals.

[0077] In residual connections, the input is directly added to the output of the transformation path:

[0078] F out =concat(F in (F2) (16)

[0079] This skip connection passes the input to the convolutional layer. It ensures that the network does not lose slowly changing low-frequency information. Meanwhile, the convolutional layer primarily learns residual patterns, i.e., high-frequency nonlinear features. The residual unit can be summarized as:

[0080] F out =(F in Conv 3×3 (σ(Conv 3×3 (σ(F in (17)

[0081] This method ensures that each unit contributes to improving the spatiotemporal representation, rather than learning the complete transformation from scratch, thereby improving the efficiency and robustness of training.

[0082] To capture heterogeneous temporal patterns at different resolutions, the input data was divided into three equal segments:

[0083]

[0084] Therefore, the input tensor is divided into:

[0085] X = [X] (w) ,X (d) ,X (r) (19)

[0086] in Capture the weekly pattern. Capture daytime mode, The encoding is dynamic in real time.

[0087] LSTM Encoding Module: While the primary temporal patterns are captured by a multi-scale residual encoder, an LSTM branch is further introduced to model sequence dependencies. This branch enhances the model's ability to learn long-term dependency features. For a given historical sequence X, it is first expanded into a node-pair-level sequence:

[0088] This involves constructing a univariate time series of length T for each OD corresponding to (i, j). This is then input into the LSTM network branch.

[0089] h (i,j),t ,c (i,j),t =LSTM(x (i,j),t ,h (i,j),t-1 ,c (i,j),t-1) (twenty one)

[0090] In the formula, h (i,j),t and c (i,j),t These are the hidden state and the unit state, respectively. This invention utilizes the final hidden state as the temporal representation of each OD pair. To align with spatial features, all hidden states corresponding to (i, j) are rearranged into tensors:

[0091]

[0092] Here, H is the hidden dimension of the LSTM. Then, a 1×1 convolutional layer maps H to a dimension consistent with the number of GPT channels:

[0093]

[0094] This branch can model the temporal evolution patterns of OD pairs, effectively capture long-term dependencies, compensate for the shortcomings of residual networks in time series modeling, and thus improve the overall prediction performance of STGM-LLM.

[0095] Two-node embedding: In OD demand forecasting, each entry point OD matrix This represents the passenger flow from the origin station to the destination station j. Since the characteristics of the origin and destination stations may differ, OD data modeling must explicitly distinguish whether each station is an origin or a destination. To this end, a dual-node embedding mechanism is designed, assigning separate embedding vectors to stations that function as both origins and destinations. Let N be the number of stations and d be the embedding dimension. Two trainable embedding matrices are introduced:

[0096]

[0097] in Encode the specific representation of the starting point of station i. The destination-specific representation of site j is encoded. For the OD pair (i, j), the embedding is constructed by concatenation:

[0098]

[0099] This operation forces the use of direction representation, where e ij ≠e ji This ensures that the model can distinguish between trips from i to j and trips from j to i, even if the same stations are involved.

[0100] After concatenation, the embeddings are further mapped to a unified latent space using two-dimensional convolution:

[0101]

[0102] Where W is a 1×1 convolution kernel, b is the bias term, and C is the input dimension of GPT. This convolution operation preserves the spatial arrangement of the OD matrix, enabling the model to capture the relationships between OD pairs.

[0103] Sparse Attention Large Model Module: LLM has achieved significant results in traffic prediction. This invention introduces a pre-trained GPT-2 as the core structure in the model backbone to enhance prediction robustness. Compared with the original GPT-2, this invention makes improvements in two aspects: first, it adopts partial frozen attention to enhance the model's robustness; second, it designs a sparse attention mechanism based on the prior sparse structure features of the OD matrix, the specific structure of which is as follows. Figure 4 As shown.

[0104] Because the amount of HSR OD data is relatively limited compared to large-scale corpora, fully training all parameters of GPT-2 can easily lead to overfitting and high computational costs. However, traffic flow prediction tasks often rely on capturing short-term fluctuations and long-term trends. Excessive parameter freezing can limit the model's adaptability. To overcome this limitation, a partial freezing attention strategy can be adopted to improve the accuracy of high-speed rail OD prediction while preserving the knowledge encoded in the pre-trained model. The core idea is selective freezing: the first few layers remain fixed, while only the multi-head attention module in the last U layer is unfrozen. The motivation for this design is the observation that higher attention layers are more flexible in capturing local and global dependencies in the input sequence, thus better simulating the dynamic spatiotemporal characteristics of passenger flow. With this design, the feedforward layers remain frozen, while the selected attention modules are trainable. This allows the model to inherit the general representations learned in large-scale pre-training while adapting to the inherent spatial heterogeneity of traffic data. Especially for high-speed rail OD prediction, when spatial structure and inter-station interactions strongly influence traffic patterns, this partial freezing mechanism provides a balance: effectively learning spatial relevance by utilizing the adaptability of unfrozen attention while preserving the potential stability of the pre-trained model. This improves the model's generalization performance.

[0105] Specifically, GPT-2 consists of 12 stacked Transformer decoder blocks. Each layer comprises a masked multi-head self-attention mechanism, a feedforward neural network (FFN), layer normalization, and residual connections. The parameter set of layer l is denoted as θ. (l) Regular fine-tuning includes updating all parameters:

[0106] Θ={Θ (1) ,Θ (2) ,...,Θ (L)} (27)

[0107] Partially frozen attention achieves a "stabilize first, then activate" learning strategy by differentially training the parameters of different blocks. Let the parameter set of each layer be represented as:

[0108]

[0109] In the formula, For self-attention parameters; These are the parameters of the feedforward network; For layer normalization parameters; Other parameters. L also represents the number of block layers used in the model.

[0110] Define the layer membership function for parameter θ as l(θ). Θ PE These are global parameters. If θ∈Θ PE Then l(θ) = 0 indicates position embedding. Define the training state function ρ(θ) ∈ {0, 1} for the parameters, where ρ(θ) = 1 indicates the parameters are trainable, and ρ(θ) = 0 indicates the parameters are frozen. Therefore, we obtain:

[0111]

[0112] This scheme allows for trainable position embeddings, enabling the model to better adapt to the feature distribution of the target domain. In the final U layer, the feedforward sublayer is frozen to mitigate overfitting, while in the preceding LU layers, all parameters remain fixed except for the layer normalization parameters, which maintain trainability to dynamically calibrate the input feature distribution during transmission.

[0113] In the OD demand forecasting task, the sequence length is determined by the number of station pair combinations N. 2 The decision and attention mechanism must be in a size of O(N) 4 The training process operates on the weight matrix of the GPT-2 self-attention module. This not only incurs significant computational overhead but also overemphasizes statistically insignificant sparse OD pairs during training, thereby reducing prediction accuracy. To address this issue, sparsity prior constraints are incorporated into the GPT-2 self-attention module. By imposing sparsity information on the attention weight matrix, this model effectively reduces redundant interactions, improving generalization ability and computational efficiency.

[0114] In standard GPT-2, the self-attention mechanism applies to the input sequence. The calculation is as follows:

[0115]

[0116] In the formula, Let query, key, and value represent these respectively, and h be the number of headers.

[0117] Based on this, embodiments of the present invention introduce a sparse prior matrix S∈[0,1]N×N S ij This represents the interaction strength between sequence positions i and j. To avoid S... ij When the logarithm is equal to 0, data overflow occurs. Sparse logarithmic weights are defined as follows:

[0118]

[0119] Finally, the sparse attention scoring function was modified as follows:

[0120]

[0121] After SoftMax normalization, the attention output is obtained as follows:

[0122]

[0123] This method employs additive constraints in the logarithmic space of the sparse matrix to directly enhance or suppress the attention scores of OD pairs, thereby guiding the model to explicitly favor statistically significant OD pairs. This is particularly meaningful in sparse OD data scenarios, where the model avoids overfitting to collateral noise relationships and maintains structural robustness through sparsity priors.

[0124] Gaussian Mixture Regression Module: Due to the existence of ticket refund and rescheduling mechanisms, high-speed rail passenger flow inherently exhibits high uncertainty. Relying solely on point estimation leads to the loss of valuable uncertainty information. To address this issue, this invention employs a Gaussian mixture layer output header, which models the conditional probability distribution, thereby capturing multimodal and heteroscedastic characteristics. Specifically, the Gaussian layer uses a finite number of components to approximate a multimodal distribution, achieving multimodal modeling, while directly outputting the probability distribution for convenient confidence interval calculation.

[0125] At each spatial location (I,j), the input x is represented by K Gaussian components: the average value μ p,k,i,j Standard deviation σ p,k,i,j >0, mixed weight π p,k,i,j Subject to constraints

[0126]

[0127] Based on Gaussian regression, two-dimensional convolution is applied to reshape the input data, thereby enabling parallel prediction and extraction of all component parameters at each spatial location.

[0128] Loss function: The output layer employs a Gaussian mixture model to model the complex and multimodal demand distribution. The training objective uses NLL loss to maximize the likelihood of the observed data under the predicted distribution. Furthermore, a weighted average absolute error (MAE) term is introduced as an auxiliary loss to improve training stability and accelerate convergence.

[0129] The training objective of this model is to make the observed OD flow y ij Maximize the likelihood function under the predicted distribution. Calculate the Gaussian log-likelihood term for each component:

[0130]

[0131] In the formula, B represents the batch size. Directly calculating the above sum can lead to numerical instability. Extremely high component likelihoods may cause exponential overflow, while extremely low component likelihoods may cause the total to approach zero, resulting in underflow. To address this issue, the log-sum-exp technique is applied for stable calculation. The log-likelihood of the mixed distribution of each sample is:

[0132]

[0133] Finally, the loss function is:

[0134]

[0135] Where β is the balance coefficient, and p represents the number of future time steps predicted by the model. This represents the actual OD (Original Discharge) flow value from origin station i to destination station j in the b-th batch. This represents the predicted OD flow value from origin station i to destination station j in the b-th batch.

[0136] IV. Model Evaluation

[0137] This invention validates its predictive performance on two real-world datasets. The following sections first introduce the datasets used, then describe the model's parameter settings and evaluation metrics. Next, several standard passenger flow prediction benchmark models are introduced, and finally, the experimental results are analyzed from multiple perspectives.

[0138] 1. Dataset

[0139] High-speed rail data. HSR0206 and HSR0703 are two real-world origin-destination (OD) passenger flow datasets for the Beijing-Shanghai High-Speed ​​Railway. This line is 1318 kilometers long and has 24 passenger stations along its route. It connects the Bohai Rim and Yangtze River Delta economic zones. The experiment used eight consecutive weeks of operational data, collected daily from 07:00 to 24:00 under train timetable constraints. Considering the long intervals between high-speed trains, the departure interval was set to 60 minutes. Eighteen OD flow matrices of size 24×24 were generated daily, resulting in a total of 1008 observation samples.

[0140] Metro data. MetroBJ2016 is the 2016 Beijing Metro system OD passenger flow dataset. This dataset covers 25 working days over five consecutive weeks, with data collection from 05:00 to 23:00 during metro operating hours. Considering the short train intervals and high frequency of passenger flow fluctuations, the time interval was set to 30 minutes.

[0141] Table 1 provides a system summary of the three datasets. All data were normalized to the range [0,1] using the min-max method.

[0142] Table 1: Data Description

[0143]

[0144] 2. Model Configuration and Evaluation Metrics

[0145] In this invention, all experiments were conducted on a workstation equipped with a 16-core Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz, 120GB RAM, and an RTX 4090 GPU (24GB), using PyTorch. The batch size was set to 8, and training was performed using the AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.0001. In the loss function, α was set to 0.3. The model received 10 time-step inputs from weekly, daily, and real-time modes to predict a single future time step. The GPT-2 architecture was configured with three layers, with three Gaussian components in each Gaussian layer. All datasets were chronologically divided into training, validation, and test sets. In the HSR dataset, the training set spanned five weeks, while the validation and test sets each covered one week. In the Metro dataset, the training set spanned three weeks, while the validation and test sets each covered one week.

[0146] Four metrics were used to evaluate model performance: root mean square error (RMSE), mean absolute error (MAE), weighted average absolute percentage error (WMAPE), and continuous ranking probability score (CRPS). MAE and RMSE quantify absolute deviation, while WMAPE measures relative error. CRPS is a comprehensive measure of probabilistic forecasting used to assess the quality of uncertain predictions. Lower values ​​for all metrics indicate higher prediction accuracy.

[0147]

[0148] in, It is an indicator function that equals 1 if x ≥ z, and 0 otherwise.

[0149] 3. Baseline Model

[0150] To verify the advantages of the model of this invention, STGM-LLM was compared with the following 11 baselines.

[0151] 1) Deterministic Model

[0152] 2D-CNN: The convolutional layers use 8 filters with 3×3 kernels and ReLU activation functions. The learning rate of this model is 0.001, and the batch size is 16.

[0153] 3D-CNN: The Conv3D layer uses 8 filters with 3×3×3 kernels and ReLU activation. This model typically uses a learning rate of 0.001 and a batch size of 16.

[0154] ConvLSTM: The ConvLSTM model is a spatiotemporal sequence network that combines convolutional and LSTM mechanisms. This model has two layers with 16 and 32 filters respectively. The kernel size is 3×3. The learning rate is 0.001, and the batch size is 16.

[0155] ConvGRU: ​​The ConvGRU model is a convolutional variant of GRU designed for spatiotemporal sequence modeling. This model has two layers with 32 and 64 filters respectively. The kernel size is 3×3. The learning rate is 0.001, and the batch size is 16.

[0156] Transformer: The Transformer employs multi-head self-attention to extract short-term and long-term flow features. The model is configured with eight parallel attention heads, each with a size of 32. Both the encoder and decoder consist of three stacked layers.

[0157] GAT: GAT assigns learnable attention weights to neighbor pairs and aggregates features using these weights. This produces a structure-aware graph convolution and enriches the node representations. We use eight parallel heads; each head has a 64-dimensional hidden vector.

[0158] 2) Probability Model

[0159] DeepAR: DeepAR uses an autoregressive LSTM to model time dependencies and adds a Gaussian output layer to quantify prediction uncertainty. This model has two layers and a size of 32.

[0160] TimeGrad: TimeGrad is an autoregressive probabilistic predictor that combines an RNN with a denoising diffusion network. This model has two layers and a size of 64.

[0161] 3) LLM class

[0162] ST-LLM: ST-LLM is built on top of the GPT-2 architecture and integrates spatiotemporal embedding with a partially frozen attention (PFA) strategy, specifically designed for traffic prediction. The model uses the first eight layers of GPT-2, with only the top two layers unfrozen to allow for task-specific adaptation.

[0163] GATGPT: GATGPT is a spatiotemporal prediction framework that combines graph attention networks with GPT-2. The model employs two graph attention layers and utilizes the first six layers of GPT-2.

[0164] OFA: OFA is also a spatiotemporal prediction model built on GPT-2. This model retains the first six layers of GPT-2.

[0165] V. Results Analysis

[0166] 1. Overall Analysis

[0167] The comparison results with the baseline model are shown in Table 2 and Figure 5 As shown.

[0168] Table 2: Performance metrics of different models on the high-speed rail dataset

[0169]

[0170] Both 2D-CNN and 3D-CNN models performed well on both datasets and exhibited similar characteristics. Interestingly, the simpler CNN model even outperformed ConvLSTM and ConvGRU on several metrics, suggesting that higher model complexity does not always lead to better results. This may be because the temporal patterns in the OD flow matrix are predominantly periodic, and the complex recurrent structures in ConvLSTM and ConvGRU may overfit short-term noise, making the fewer-parameter CNN more stable on small to medium-sized datasets. Furthermore, the Transformer and GAT models performed poorly. Notably, GAT's performance dropped significantly on the High-Speed ​​Rail (HSR) data. This may be because the graph topology in the HSR network fails to accurately capture the true connections between stations—GAT allocates attention based on adjacency relationships, often overemphasizing adjacent nodes while ignoring distant but highly correlated OD pairs, resulting in a larger average error.

[0171] As probabilistic models, DeepAR and TimeGrad demonstrate good uncertainty calibration capabilities, but they focus more on modeling the overall distribution rather than achieving precise point predictions, resulting in lower point prediction accuracy compared to deterministic models. Among baseline models based on Large Language Models (LLM), the overall trend shows that LLM has certain advantages in spatiotemporal OD modeling, but differences in model structure lead to varying performance: ST-LLM performs significantly better; GATGPT, although incorporating a graph attention mechanism, is still limited by the inherent problems of GAT, resulting in limited improvement; OFA, as a general multimodal model, performs relatively poorly due to the low adaptability of its pre-training objectives to traffic-related tasks. In contrast, the proposed STGM-LLM achieves the best overall performance on both datasets, its advantage mainly stemming from its structured input design, which integrates matrix sparsity features and a two-node embedding mechanism.

[0172] Overall, STGM-LLM not only outperforms all probabilistic models in both deterministic and probabilistic metrics, but also achieves point prediction accuracy comparable to deterministic methods.

[0173] 2. Analysis of a single OD pair

[0174] To evaluate the predictive performance of the model, several representative OD pairs were selected from the two datasets, and the actual values ​​were compared with the predicted values, such as... Figure 6 and Figure 7 As shown.

[0175] In the two high-speed rail datasets, STGM-LLM accurately captured the temporal changes in passenger flow. Figure 6 Three high-demand outer diameter (OD) pairs, ranging from shortest to longest trip distances, are presented. The results show that long-distance OD pairs exhibit more pronounced periodicity due to the higher regularity of long-term travel planning. In contrast, short-distance OD pairs show stronger fluctuations in subsequent time steps, especially around weekends (Friday to Sunday), reflecting more spontaneous short-term trips. Regardless of distance, the model performs well, with its 95% confidence interval demonstrating strong robustness. Figure 7 The model further illustrates a high-demand OD pair and two low-demand OD pairs. It demonstrates that STGM-LLM accurately captures flow dynamics across different demand levels. Even for sparse and highly volatile OD pairs, the model maintains stable predictions, highlighting its strong generalization and robustness. This advantage primarily stems from the sparse attention mechanism, which allows the network to focus on informative OD relationships while suppressing noise from irrelevant flows.

[0176] In summary, the proposed STGM-LLM demonstrates robust and reliable predictive performance across different transportation systems. It achieves high accuracy not only in high-speed rail OD prediction but also in metro OD prediction, proving the adaptability and generalization of this method across different transportation modes.

[0177] 3. Extensibility Analysis

[0178] To verify the scalability of the model, it was applied to the Beijing Metro OD dataset. The results are as follows: Figure 8 As shown in Table 3.

[0179] Table 3: Metrics of different models on the subway dataset

[0180]

[0181] Overall, the proposed model also demonstrates strong performance on the metro dataset, indicating excellent scalability. It is noteworthy that although the model achieves better prediction accuracy, the improvement on the best baseline is less than the improvement observed on the high-speed rail dataset. There are three main reasons for this difference. First, the OD matrix of the high-speed rail system is much sparse because passenger flow is typically concentrated at a few hub stations, while many other stations have very low passenger volumes. The large proportion of zero or near-zero entries creates a highly imbalanced training objective. Second, the greater spatial distance between high-speed rail stations leads to weaker spatial correlations. In contrast, adjacent stations in the metro network are more tightly coupled, and their spatiotemporal patterns are more regular. This explains why some locally focused models perform better on metro data. Finally, high-speed rail passenger flow is more random than metro passenger flow. Metro travel is mainly commuting and daily activities, exhibiting strong periodicity. In contrast, high-speed rail travel is more influenced by holidays and occasional long-distance trips, which occur less frequently and irregularly. Therefore, the model performs very well in capturing the inherent patterns in metro passenger flow data.

[0182] Similarly, several representative OD pairs were selected from the subway dataset, and the actual values ​​were compared with the predicted values, such as... Figure 9 As shown.

[0183] The cyclicality of subway passenger flow is significantly stronger than that of high-speed rail passenger flow. This is consistent with the previous analysis. This regularity can be attributed to three main factors. First, subway data is collected at finer-grained time intervals to accurately capture short-term fluctuations. Second, subway travel follows different daily and weekly cycles, providing clear temporal patterns. Third, the subway network operates within compact urban areas, with frequent services and closely connected stations, making the spatial correlation between origin-destination (OD) pairs more consistent.

[0184] 4. Ablation Experiment Analysis

[0185] The effectiveness of each component in the proposed model was further verified through ablation experiments. The variant model used for comparison is described below:

[0186] w / o LSTM(V1): Remove the LSTM module to verify the mechanism's functionality.

[0187] w / o real-time ResNet(V2): Removes real-time mode input from the residual network.

[0188] w / o daily ResNet(V3): Removes the daily pattern input from the residual network.

[0189] w / o weekly ResNet(V4): Removes weekly pattern inputs from the residual network.

[0190] w / o ResNet(V5): Remove the entire residual network module.

[0191] w / o sparse attention(V6): Replaces the sparse attention mechanism with the standard attention mechanism.

[0192] w / o GPT-2(V7): Replaces the GPT-2 module with a lightweight class GPT Transformer to reduce model complexity.

[0193] w / o dual-node embedding(V8): Removes the dual-node embedding module.

[0194] STGM-LLM: The complete version of the proposed model.

[0195] Table 4: Predictive performance of all variant models on the three datasets

[0196]

[0197] Ablation experiments clearly demonstrate that each module contributes differently to the STGM-LLM framework. Among all variants, removing the two-node embedding (V8) leads to a sharp drop in performance on the HSR dataset. This confirms that explicitly encoding the start and end points, and their spatial relationships, is crucial for high-speed rail scenarios with long station-to-station distances. The residual network modules (V2-V5) play a complementary role by capturing real-time, daily, and weekly patterns. Removing any of them reduces accuracy. When the entire residual component is removed, model performance drops significantly, indicating that the residual network is crucial for point prediction and calibration uncertainty. This is because the ResNet layers extract stable periodic components, allowing the LLM backbone to focus on the remaining fluctuations. Without them, the backbone must handle all dynamics simultaneously, which compromises accuracy. Replacing sparse attention with standard attention (V6) also degrades performance. It diverts focus from meaningful OD pairs and introduces noise from weakly correlated flows, particularly in sparse HSR matrices. Interestingly, replacing the GPT-2 backbone with a lightweight Transformer (V7) leads to a significant decrease in the probability metric (CRPS), although the deterministic error (RMSE) remains similar. This suggests that GPT-2's advantage lies not only in sequence modeling but also in capturing rich contextual dependencies.

[0198] 5. Hyperparameter Analysis

[0199] STGM-LLM involves two hyperparameters that can potentially affect model performance. Therefore, the impact of these hyperparameters on model performance was analyzed using the HSR0206 dataset, and the results are as follows: Figure 10 As shown.

[0200] The impact of the number of GPT-2 layers (L): The number of GPT-2 layers determines the model size, directly affecting its spatiotemporal modeling capabilities and training time. As the number of layers increases from 2 to 3, model performance gradually improves, reaching its optimal performance at L=3. Beyond this value, various metrics begin to fluctuate or even decline. A 2-layer GPT-2 structure produces higher root mean square error (RMSE) and mean absolute error (MAE), indicating that shallow models cannot fully capture the complex dynamic features and long-term dependencies in the OD spatiotemporal sequence, and their limited semantic modeling capabilities lead to larger prediction errors. When the number of layers increases to 3 or 4, the model achieves a balance between temporal abstraction capabilities and contextual understanding. However, when the depth exceeds 4 layers, RMSE, MAE, and weighted average absolute percentage error (WMAPE) all show a slight increase; although the Continuous Ranking Probability Score (CRPS) improves slightly at 8 layers, deeper architectures do not bring overall performance improvements. This is mainly because excessively deep networks increase training difficulty, leading to decreased gradient propagation stability, and are prone to overfitting under limited OD data conditions.

[0201] The impact of the number of Gaussian mixture distributions (K): In the STGM-LLM framework, the number of Gaussian distributions controls the flexibility of the model in capturing the probability distribution of OD demand. The model performs best when K=3, achieving a good balance between complexity and robustness. When K is too small, the model cannot describe the multimodal or heteroscedastic patterns in OD flow, leading to increased RMSE and WMAPE; when K is too large, the indicators deteriorate again, indicating that too many mixture parameters can cause the model to overfit local noise with limited samples, thus impairing overall performance. This result shows that the choice of the number of mixture components affects both the uncertainty estimation effect and the point prediction accuracy.

[0202] 6. Performance analysis with few samples

[0203] In the few-shot prediction task, STGM-LLM was trained using only one week's worth of data, and the relevant results are shown in Table 5.

[0204] Table 5: Prediction Results for Small Samples

[0205]

[0206] The results show that the model of this invention consistently achieves the lowest mean absolute error (MAE) and weighted average absolute percentage error (WMAPE) across all three datasets, with superior root mean square error (RMSE). This advantage primarily stems from the pre-trained knowledge of the GPT-2 backbone network, specifically the temporal dependencies and attention decomposition capabilities learned during the pre-training phase. This allows the model to capture complex spatiotemporal patterns even with extremely limited data. Furthermore, the pre-trained attention layer serves as a structural prior, helping the model maintain stability when handling non-stationary OD flows. In contrast, while ST-LLM can extract spatiotemporal information, it lacks specific processing for sparse OD data, leading to biased predictions of many low-flow OD pairs, which in turn increases MAE and WMAPE. OFA and GATGPT exhibit a characteristic of "low RMSE but high MAE and WMAPE": this indicates that these models perform well on high-flow OD pairs (thus reducing RMSE), but perform poorly on a large number of low-flow OD pairs. The frequent relative errors in these low-flow pairs ultimately result in higher overall MAE and WMAPE.

[0207] In summary, compared with the prior art, the present invention has the following main advantages:

[0208] (1) This invention proposes a multi-scale OD time series decomposition method, which decomposes the original high-speed rail OD sequence into a weekly cycle sequence, a daily cycle sequence, and a real-time fluctuation sequence according to the operation pattern, and processes passenger flow patterns under different time scales respectively, effectively enhancing the model's ability to perceive multiple time series structures. In addition, this invention introduces a sparse attention prior matrix construction method based on OD structure, which uses information such as OD long-tail distribution, station association strength, or distance to generate sparse structure priors, reducing redundant structures in OD input from the source and improving the effectiveness of the model under high-dimensional sparse input.

[0209] (2) This invention proposes a spatial convolution-temporal series dual-branch feature extraction module (ResNet+LSTM) to extract spatial neighborhood association features and long-sequence temporal dependency features from the OD matrix, achieving simultaneous expression of the spatiotemporal structure of OD and solving the problem that traditional methods can only capture a single structure. Furthermore, this invention proposes a priori-based sparse self-attention mechanism, introducing the sparse structure of OD into the Transformer attention calculation, reducing the interference of invalid attention terms, and achieving structural adaptation capability for long-tailed OD data. In addition, this invention proposes a probability output module based on a Gaussian mixture model (GMM), enabling the model to simultaneously output the mean, standard deviation, and multimodal probability distribution of OD predictions, structurally achieving uncertainty prediction and improving the ability to express the randomness caused by actions such as ticket changes, refunds, and waiting lists.

[0210] (3) This invention proposes a probability distribution training method based on negative log-likelihood (NLL), which directly performs maximum likelihood training on the output of the Gaussian mixture model, enabling the model to learn the shape, fluctuation range, and uncertainty structure of the future OD distribution. Furthermore, this invention proposes an auxiliary training objective based on weighted average absolute error (MAE), which improves the accuracy and stability of the point prediction part by simultaneously constraining the prediction mean, thus solving the mean bias problem that may result from relying solely on probability training.

[0211] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0212] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0213] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0214] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0215] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for predicting high-speed rail origin-destination (OD) passenger flow based on a large language model, comprising the following steps: For the target site, obtain historical OD sequences at multiple time steps; Based on the historical OD sequence, the OD passenger flow prediction results for future time steps are obtained using a trained large language model. The large language model includes an input data processing module, a spatiotemporal feature extraction module, a sparse attention large model module, and a Gaussian mixture prediction module. The input data processing module is used to divide the historical OD sequence into temporal components of different modes. The spatiotemporal feature extraction module is used to capture the spatial and temporal dependencies of the historical OD sequence. The sparse attention large model module is used to obtain attention output based on a sparse prior matrix. The Gaussian mixture prediction module is used to approximate a multimodal distribution with a finite number of components to achieve multimodal modeling and then output a probability distribution.

2. The method according to claim 1, characterized in that, The empty feature extraction module includes a residual module, an LSTM encoding module, and a two-node embedding module. The residual module is used to extract high-level spatiotemporal features corresponding to the time components of the different time patterns. The LSTM encoding module is used to model the order dependency of the historical OD sequence. The dual-node embedding module is used to capture the relationship between OD pairs.

3. The method according to claim 2, characterized in that, The time components of the different modes include weekly mode time components, daily mode time components and real-time mode time components. The residual module contains three residual units, and each time component corresponds to one mode of time component processing.

4. The method according to claim 2, characterized in that, The LSTM encoding module models the order dependency of the historical OD sequence according to the following steps: For a given historical OD sequence X, it can be expanded into a node-pair sequence: With x ij As input to the LSTM encoding module, the hidden state h is obtained. (i,j),t and unit state c (i,j),t : h (i,j),t ,c (i,j),t =LSTM(x (i,j),t ,h (i,j),t-1 ,c (i,j),t-1 Rearrange all hidden states corresponding to (i, j) into a tensor: H is mapped to a dimension consistent with the number of channels in the sparse attention large model module using a 1×1 convolutional layer: Where T is the length of the node pair sequence, H is the hidden dimension of the LSTM encoding module, and N is the number of stations.

5. The method according to claim 2, characterized in that, The dual-node embedding module performs the following embedding mechanism: Construct two trainable embedding matrices: in Encode the specific representation of the starting point of station i. The destination-specific representation of site j is encoded, and for OD pairs (i,j), the embedding is constructed by concatenation: After concatenation, the embeddings are mapped to a unified latent space using two-dimensional convolution: Where W is a 1×1 convolution kernel, b is the bias term, C is the input dimension, N is the number of stations, d is the embedding dimension, and e is the input dimension. ij ≠e ji .

6. The method according to claim 1, characterized in that, The sparse attention large model module obtains the attention output based on the following formula: in: The sparse attention scoring function is: Among them, S ij This represents the interaction strength between sequence positions i and j, where Q, K, and V represent the query, key, and value, respectively, ∈ is a constant, and d k It is the scaling factor.

7. The method according to claim 1, characterized in that, The training process of the large language model adopts a selective attention freezing strategy, keeping the first few layers fixed and unfreezing the multi-head attention module in the last U layer.

8. The method according to claim 1, characterized in that, The loss function for training the large language model is set as follows: in: in, This represents the negative log-likelihood loss term. represents the mean absolute error loss term, β is the balance coefficient, B is the batch size, N is the sample size, and p represents the number of future time steps to be predicted. This represents the actual OD (Original Discharge) flow value from origin station i to destination station j in the b-th batch. This represents the predicted OD flow value from origin station i to destination station j in the b-th batch.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images