A traffic flow prediction method based on federal inversion Transformer

Abstract

The application discloses a traffic flow prediction method based on a federal inversion Transformer, relates to a federal learning technology, and is proposed in view of problems in the prior art. In a federal learning architecture composed of K clients and a central server, an inversion Transformer is used as a local prediction model of each client; each client only uses private traffic data locally to train the local model, and an aggregation mechanism weighted according to data scales is used to suppress the adverse effects of the clients on the global model; each client only uploads model parameters to the central server; the central server aggregates a new global model and distributes the new global model to each client, and iteration is performed until convergence. The method has the advantages that the privacy is effectively protected while the traffic flow prediction accuracy is maintained; the multivariate long-time series modeling capability is strong; the weighted aggregation strategy effectively alleviates the influence of data heterogeneity, and the global model converges more stably.

Description

Technical Field

[0001] This invention relates to federated learning technology, and more particularly to a traffic flow prediction method based on a federated inverted Transformer. Background Technology

[0002] Traffic flow prediction (TFP) is a core component of Intelligent Transportation Systems (ITS). It aims to estimate vehicle traffic patterns at various nodes in the road network over a future timeframe by modeling and analyzing historical traffic flow data. This is of significant value in alleviating traffic congestion, optimizing resource allocation, and supporting urban planning. Current mainstream traffic flow prediction methods can be divided into two main categories: one is parametric statistical models, represented by ARIMA and Kalman filtering; the other is deep learning models, represented by Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM), Gated Recurrent Units (GRU), Graph Convolutional Networks (GCN), and Transformers. Deep learning models, due to their strong modeling capabilities for nonlinear spatiotemporal characteristics, have become the mainstream technical approach for traffic flow prediction.

[0003] In practical deployments, traffic flow data originates from various sources, including roadside sensors, video surveillance, GPS, in-vehicle terminals, and navigation / ride-sharing platforms. This data is often scattered across different clients, which can be deployed in institutions, enterprises, or regions, and frequently contains sensitive personal information such as vehicle location, travel trajectories, license plates, and images of drivers and passengers. Due to privacy regulations, it is difficult for clients to directly exchange raw data; model training can only be completed locally. This results in a limited amount of training data available to a single client, restricting the model's generalization ability and prediction accuracy.

[0004] Federated Learning (FL), as a distributed machine learning paradigm, allows multiple clients to collaboratively train a shared model without sharing the original data. It aggregates knowledge from multiple parties simply by exchanging model parameters, making it a crucial technological path to resolve the data silos and privacy concerns mentioned above. In recent years, research has introduced federated learning into traffic flow prediction, such as FedGRU based on GRU and federated time series prediction schemes based on LSTM. Meanwhile, novel sequence modeling architectures, represented by iTransformer (inverted Transformer), embed each variable of a multivariate time series as a single token, allowing the attention mechanism to directly capture the correlation between variables. This results in significantly higher accuracy than traditional RNN models with long prediction step sizes.

[0005] The existing technology has the following defects and shortcomings: (1) Centralized deep learning models pose a serious risk of data privacy leakage. Centralized solutions, represented by CNN / LSTM / Transformer, require the original traffic data from all parties to be aggregated to a central server for training. Under the constraints of privacy regulations, this is difficult to implement in practice. Moreover, once the central node is attacked, it will cause large-scale leakage of user privacy.

[0006] (2) Traditional privacy protection methods come at the cost of prediction accuracy. Although traditional privacy protection methods such as differential privacy, k-anonymity, and bit-group-based mask statistics can protect privacy, they usually come at the cost of significantly reducing the accuracy of traffic flow prediction, making it difficult to balance accuracy and privacy.

[0007] (3) Existing federal traffic forecasting schemes use RNN series local models, which are insufficient in multivariate modeling capabilities. Existing federated learning traffic forecasting schemes (such as FedGRU based on GRU, LSTM based schemes, etc.) mostly use recurrent neural network (RNN) series time series forecasting models, which are insufficient in modeling long-range dependence and multivariate correlation of multivariate long time series data such as traffic flow, and the forecasting accuracy decreases significantly under long forecast step lengths.

[0008] (4) Existing federated aggregation does not fully consider the heterogeneity of traffic data, resulting in unstable global model performance. Existing federated aggregation often adopts a simple averaging of parameters for each client, which does not fully consider the data heterogeneity problems that are common among clients in traffic flow scenarios. These include differences in road network structure, sensor deployment density, sampling frequency, traffic flow tidal patterns, and unbalanced sample size, which are non-independent and identically distributed characteristics. This leads to slow convergence and increased performance fluctuations of the global model under heterogeneous data conditions. Summary of the Invention

[0009] The purpose of this invention is to provide a traffic flow prediction method based on the Federal Inverted Transformer to solve the problems existing in the prior art.

[0010] The traffic flow prediction method based on federated inverted Transformer described in this invention uses an inverted Transformer as the local prediction model for each client in a federated learning architecture consisting of K clients and a central server. Each client trains its local model using only private traffic data, and the adverse effects of the client on the global model are suppressed by an aggregation mechanism weighted by data size. Each client only uploads model parameters to the central server. The central server aggregates the data to obtain a new global model and distributes it to each client, iterating until convergence.

[0011] The traffic flow prediction method based on federated inverted Transformer described in this invention has the advantage that the original traffic flow data of each client is always stored locally, and model parameters are only uploaded to the central server. This eliminates the risk of privacy leakage caused by data aggregation and eliminates the need for post-event remedial measures such as data anonymization. At the same time, privacy protection is embedded in the federated training framework itself, without adding additional information perturbations to the output or parameters of the prediction model, thus not reducing the upper limit of the accuracy of the local prediction model. Compared with centralized schemes and traditional privacy protection methods such as differential privacy and k-anonymity, which introduce data perturbations or information loss, this invention can maintain high traffic flow prediction accuracy while effectively protecting privacy.

[0012] Using the inverted Transformer as the local prediction model for each client, the attention mechanism directly captures the spatial correlation between traffic flows across multiple road segments by embedding the complete time series of each variable as an independent token. The feedforward network learns the temporal nonlinear features within each road segment. Compared to traditional RNN models, the inverted Transformer achieves higher prediction accuracy with longer prediction step sizes and provides a more comprehensive model of multivariate traffic flow data.

[0013] On the server side, an aggregation strategy weighted by the sample size of each client is adopted. This allows clients with larger data scales and more representative sample distributions to contribute higher weights in global model updates, thereby suppressing the adverse interference of noise from small-sample clients on the global model. Compared with simple average aggregation (FedAvg), this invention achieves faster global model convergence, less performance fluctuation, and stronger generalization ability, even under conditions where traffic data from different clients exhibit non-independent and identically distributed characteristics such as differences in road network structure, sensor density, sampling frequency, and sample size imbalance. Attached Figure Description

[0014] Figure 1 This is a comparative diagram of the centralized training scheme and the federated training scheme of this invention.

[0015] Figure 2 This is a schematic diagram of the federated learning architecture of the present invention.

[0016] Figure 3 This is a schematic diagram of the federated training process of this invention. Detailed Implementation

[0017] like Figures 1 to 3As shown, the traffic flow prediction method based on federated inverted Transformer described in this invention uses an inverted Transformer as the local prediction model for each client within a federated learning architecture consisting of K clients and a central server. To address the data heterogeneity caused by inconsistencies in sample distribution, road network characteristics, and sampling conditions among different clients in traffic flow scenarios, each client trains its local model using only its own private traffic data. A data-weighted aggregation mechanism is used to suppress the adverse effects of clients with small data volumes and high noise levels on the global model. Each client only uploads model parameters to the central server, without uploading any raw data. The central server aggregates the data to obtain a new global model and distributes it, iterating until convergence.

[0018] Notation conventions: K represents the total number of clients; k represents the client ID; t represents the federation iteration round; These are the global model parameters for round t; These are the local model parameters for client k; For client k's local dataset, the number of samples is Total number of samples η is the learning rate; Let k be the local loss function for client k. Let i be the true label value of the i-th sample. Let be the predicted value for the i-th sample, where i is the index of the sample.

[0019] The overall process steps are as follows: S1. Initialization Steps: The central server initializes global model parameters. It is then distributed to all K clients; the local prediction model uses an inverted Transformer structure.

[0020] S2. Local training steps: Client k receives global parameters. Then, in local data Update local parameters using gradient descent: (Equation 1); The local loss for the prediction task is expressed in the form of mean squared error: (Equation 2).

[0021] S3. Parameter Upload Steps: Each client uploads the updated model parameters locally. Uploaded to the central server, but without uploading any original traffic data.

[0022] S4. Weighted Aggregation Step: The central server weights and aggregates the uploaded parameters according to the sample size of each client to obtain new global parameters. (Equation 3); (Equation 4); S5. Iterative convergence step: Repeat steps S2 to S4 until the preset number of rounds is reached or convergence is achieved; finally, the global model is used to predict traffic flow F(t) under different prediction step sizes.

[0023] The inverted Transformer embeds each variable, such as the entire time series of each sensor / road segment, independently as a token, allowing the attention mechanism to directly capture multivariate correlations: (Equation 5); (Formula 6); (Equation 7); in, The aggregate weight of client k satisfies F(t) represents the predicted traffic flow at time t by the global model after the federated training has converged. loss function The gradient; v is the variable number; For the complete input time series of the v-th variable; for The token representation obtained after the variable embedding layer; This is the intermediate representation after processing by the multi-head self-attention layer; This is the hidden layer representation after processing by the feedforward network layer.

[0024] Unlike traditional Transformers that segment tokens along the time axis, the inverted Transformer segments tokens along the variable axis, treating the complete time series of each variable as a token. An attention mechanism learns the spatial correlations between variables, while a feedforward network learns the temporal nonlinear characteristics within each variable. This allows each client to effectively model the collaborative changes in traffic flow across multiple road segments locally, maintaining high prediction accuracy even at long prediction step sizes, such as 96 / 192 / 336 time steps.

[0025] To address the heterogeneity of traffic data from different clients in terms of road network structure, sensor density, sampling frequency, and sample size, the central server adopts a weighted aggregation strategy based on sample size (Equations 2 to 3). This allows clients with larger sample sizes and more stable data quality to contribute more weight in global model updates, thereby suppressing noise interference from small sample clients and mitigating the global model shift problem caused by simple average aggregation under non-independent and identically distributed conditions.

[0026] To implement the method, this invention provides a federated learning architecture, specifically composed of the following: Data localization module: Used for each client to store private traffic flow data locally, without sharing the original data with external parties.

[0027] Local training module: Used by each client to perform gradient descent training on local data using the inverted Transformer as the prediction model (Equation 1).

[0028] Parameter upload module: Used by each client to upload the updated model parameters to the central server.

[0029] Weighted aggregation module: used by the central server to calculate the aggregation weight based on the sample size of each client (Equation 3), and perform weighted aggregation (Equation 4) to obtain a new global model.

[0030] Model distribution and iteration module: This module is used by the central server to distribute new global models to each client, driving the next round of iteration.

[0031] Prediction output module: Used to perform traffic flow prediction inference on each client local after the global model has converged, and output the future multi-step prediction result F(t).

[0032] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A traffic flow prediction method based on a federated inverted Transformer, characterized in that, In a federated learning architecture consisting of K clients and a central server, an inverted Transformer is used as the local prediction model for each client. Each client trains its local model using only private traffic data. An aggregation mechanism weighted by data size is used to suppress the adverse effects of clients on the global model. Each client only uploads model parameters to the central server. The central server aggregates the data to obtain a new global model and distributes it to each client, iterating until convergence.

2. The traffic flow prediction method based on the Federal Inverted Transformer as described in claim 1, characterized in that, Includes the following steps: S1. Initialization Steps: The central server initializes global model parameters. And it is distributed to all K clients; the local prediction model uses an inverted Transformer structure; S2. Local training steps: Client k receives global parameters. Then, in local data Update local parameters using gradient descent: ； The local loss for the prediction task is expressed in the form of mean squared error: ； S3. Parameter Upload Steps: Each client uploads the updated model parameters locally. Uploaded to the central server, without uploading any original traffic data; S4. Weighted Aggregation Step: The central server weights and aggregates the uploaded parameters according to the sample size of each client to obtain new global parameters. ； ； S5. Iterative convergence step: Repeat steps S2 to S4 until the preset number of rounds is reached or convergence is achieved; the final global model is used to predict traffic flow F(t) under different prediction step sizes; in, K represents the total number of clients; k is the client ID; t is the federation iteration round; These are the global model parameters for round t; These are the local model parameters for client k; This is the local dataset for client k; the number of samples is... Total sample size η is the learning rate; Let k be the local loss function for client k. The aggregate weight for client k; loss function The gradient; Let i be the true label value of the i-th sample. Let be the predicted value for the i-th sample, where i is the index of the sample.

3. The traffic flow prediction method based on the Federal Inverted Transformer according to claim 2, characterized in that, The inverted Transformer embeds each variable independently as a token, allowing the attention mechanism to directly capture multivariate correlations: ； ； ； Where v is the variable number; For the complete input time series of the v-th variable; for The token representation obtained after the variable embedding layer; This is the intermediate representation after processing by the multi-head self-attention layer; This is the hidden layer representation after processing by the feedforward network layer.

4. The traffic flow prediction method based on the Federal Inverted Transformer according to claim 3, characterized in that, This is achieved through a federated learning architecture, which consists of the following components: Data localization module: Used for each client to store private traffic flow data locally, without sharing the raw data externally; Local training module: Used by each client to perform gradient descent training on local data using the inverted Transformer as the prediction model; Parameter upload module: Used by each client to upload the updated model parameters to the central server; Weighted aggregation module: used by the central server to calculate the aggregation weight based on the sample size of each client, and perform weighted aggregation to obtain a new global model; Model distribution and iteration module: Used by the central server to distribute the new global model to each client, driving the next round of iteration; Prediction output module: Used to perform traffic flow prediction inference on each client local after the global model has converged, and output the future multi-step prediction result F(t).

Images