Attention mechanism based spatiotemporal fusion graph generation method for multi-modal traffic

Abstract

The application discloses a kind of space-time fusion graph generation methods based on attention mechanism to face multi-mode traffic, comprising the following steps: step one, obtain time graph and space graph from total traffic flow data and generate adaptive space-time fusion graph by fusion;Step two, total traffic flow is decoupled and segmented;Step three, the graph convolution is carried out to the adaptive space-time fusion graph generated by traffic mode flow after decoupled segmentation, and the space-time characteristics in each traffic mode flow are extracted;Step four, the space-time characteristics in each traffic mode flow extracted are spliced and further obtained the hidden state of traffic mode flow in its total time step by gate mechanism;Step five, the information of total traffic flow, time graph, decoupled traffic mode flow and the hidden state of the total time step is connected by jumping, to obtain multi-scale feature, then form regression layer by full connection and two-dimensional convolution, obtain traffic flow prediction result.The application effectively improves the accuracy of traffic flow prediction.

Description

Technical Field

[0001] This invention belongs to the field of traffic flow prediction technology and relates to a spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism. Background Technology

[0002] The accuracy and sophistication of traffic flow forecasting are driving the development of research in transportation planning, intelligent transportation systems, and urban sustainability. Many traditional methods focus solely on spatial proximity when dealing with spatiotemporal relationships, neglecting the potential temporal similarities between nodes. In real-world scenarios, even spatially distant nodes may exhibit significant temporal correlations. This spatiotemporal correlation manifests in diverse ways across different fine-grained traffic patterns. Furthermore, spatiotemporal relationships are typically dynamic and evolve over time. Therefore, constructing dynamic spatiotemporal fusion maps that adapt to changes at different points in time and capture the spatiotemporal relationships of different traffic modes is crucial.

[0003] However, recent research often involves constructing a spatial graph with a predefined or data-adaptive adjacency matrix, using graph neural networks to capture the correlations between spatial nodes at each time step, and using RNNs or transformers to connect each node to itself between adjacent time steps to capture information from temporally adjacent nodes. However, these methods are typically limited by static spatial modeling and fail to accurately capture the dynamic and complex relationships between time and space. Specifically, they fail to fully extract the spatiotemporal relationships between different time steps and nodes, including the dynamic changes in these relationships, thus affecting accuracy. Therefore, more effective methods are needed to extract the spatiotemporal relationships between different time steps and nodes, including the dynamic changes in these relationships, and to consider assigning different spatiotemporal fusion graphs to different traffic modes. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, this invention proposes a method for generating a spatiotemporal fusion map for traffic flow prediction based on an attention decoupling network. The specific technical solution is as follows:

[0005] A spatiotemporal fusion graph generation method based on attention mechanism for multimodal traffic includes the following steps:

[0006] Step 1: Construct a dynamic graph generation module to obtain and construct temporal and spatial graphs from total traffic flow data. Then, based on an attention mechanism, fuse the temporal and spatial graphs and obtain the correlation between them to generate an adaptive spatiotemporal fusion graph.

[0007] Step 2: Construct a decoupling module to decouple the total traffic flow into different traffic mode flows.

[0008] Step 3: Construct a residual graph convolution module to perform graph convolution on the generated adaptive spatiotemporal fusion graph of each traffic mode flow to extract the spatiotemporal features of each traffic mode flow;

[0009] Step 4: Construct a time series module. The spatiotemporal features extracted from each traffic pattern flow are spliced ​​together and input into the time series module. The hidden state of the traffic pattern flow at its total time step is further obtained through a gating mechanism.

[0010] Step 5: Construct a skip connection layer module to skip connections the total traffic flow, time map, decoupled traffic pattern flow, and the hidden state of the total time step to obtain multi-scale features. Then, a regression layer is formed through fully connected layers and two-dimensional convolution to generate a traffic flow prediction state result map.

[0011] Furthermore, the construction of the spatial graph is specifically expressed as follows:

[0012] M1 = tanh(αW1E1)

[0013] M2=tanh(αW2E2)

[0014]

[0015]

[0016] Where E1∈R N×d ,E2∈R N×d W1 and W2 represent the node space embedding matrix that is randomly initialized, W1 and W2 are the parameter matrices of the fully connected layer, and α is the parameter that controls the saturation rate.

[0017] The spatial graph A is obtained by regularizing the paired matrices M1 and M2 using an activation function and performing subtraction terms. s ∈R N×N Then the matrix The largest K elements in each row are retained, and the other elements are set to zero to obtain the adjacency matrix.

[0018] The specific expression for constructing the time graph is as follows:

[0019]

[0020]

[0021] in, Indicates the total traffic flow X in The time period matrix obtained in the first step is to reduce the dimension T. h Average pooling, for the first T h Taking the average of the step sizes reduces the spatial dimension, resulting in matrix T.D ∈R N×d ,T W ∈R N×d ;

[0022] m1 = tanh(T) D (T D ) T )

[0023] m2=tanh(T W (T W ) T )

[0024]

[0025]

[0026] By analyzing matrix T D and T W Performing the transpose dot product operation yields the similarity matrix m1∈R. N×N m2∈R N×N Then, the time-map relationship matrix is ​​generated by subtraction, thus obtaining time-map A. t ∈R N×N Then, a masking operation is performed to retain the top-k nearest node connections.

[0027] Furthermore, the spatial graph adjacency matrix A s Relationship matrix A with time graph t A multi-head attention mechanism was used to assign weights to capture the correlation between the spatial and temporal graphs during the fusion process. The formula is expressed as:

[0028] A f =ReLU(tanh(βA) s A t ))

[0029]

[0030] Multihead s (A s A t A f ) = (head1||head2||…||head s W o

[0031]

[0032] Where β is a learnable parameter, A f ∈R N×N Spatial graph A was further captured by using activation functions and scaling.s And time graph A t Interaction effects; spatial diagram A s ∈R N×N and learnable query matrix Multiply to obtain the query variable, time graph A t ∈R N×N and learnable key matrix Multiply to obtain the key variable, calculate the similarity between the query vector and the key vector, and then divide the similarity by . Scaling is performed; the similarity after scaling is further processed using the softmax function to obtain the attention weight W. o Finally, attention weight values ​​are used to evaluate key value variables. Weighting, The learnable value matrix enables the output spatiotemporal fusion graph to dynamically allocate weights by extracting the similarity relationships between spatiotemporal graphs obtained from different traffic modes.

[0033] Furthermore, step two specifically involves: using a node space embedding matrix initialized with learnable parameters to capture the semantic information and feature representation of each node in the traffic flow network; performing nonlinear transformation and feature extraction on the features through a fully connected layer and the modified linear activation function ReLU; and using the Sigmoid activation function to generate a decoupling module value to estimate the proportional relationship between different traffic mode flows; then, multiplying the decoupling module value element-wise with the total traffic flow data to obtain different traffic mode flows.

[0034] Furthermore, the formula for the decoupling module used to separate the total traffic flow into two traffic modes is expressed as follows:

[0035]

[0036] X1 = X in ⊙Ω

[0037] X2 = X in -X1

[0038] Wherein, the weight matrix W1∈R 3d×d W2∈R d×1 A linear transformation is performed on the feature vector, and nonlinear activation functions ReLU and Sigmoid are introduced to obtain Ω. t,i ∈(0,1), representing the proportion of a specific traffic pattern at node i relative to the total traffic flow at time step t; matrix Represents a specific traffic mode relative to the total traffic flow in T h The output, representing the proportion at each time step and N nodes, is broadcast via Hadamard product to... All channels; via the decoupling proportional matrix Ω and total traffic flow Xin Perform element-wise multiplication to obtain a partial traffic flow. Indicates the contribution of a specific transportation mode; subsequently, through X in Subtracting the aforementioned traffic flow X1 from the traffic flow yields another traffic flow X2, which contains flow not explained by a specific traffic pattern.

[0039] Furthermore, the formula for graph convolution performed by the residual graph convolution module in step three is expressed as follows:

[0040]

[0041]

[0042] h out =(h (0) ||h (1) ||…||h (k-1) W

[0043] Where A represents the adjacency matrix. It is an adjacency matrix with added self-connections I, and a degree matrix. And satisfy k represents the propagation depth, W represents the convolutional layer weights, and h... in and h out These represent the outputs of the previous and current residual graph convolutional layers in the residual graph convolutional module, respectively. The parameter α indicates the retention of input node h during node updates. in The proportion.

[0044] Furthermore, the formula for spatiotemporal feature splicing in step four is expressed as follows:

[0045]

[0046]

[0047] Where N is the number of iterations in the residual graph convolutional module RGC, P is the number of decouplings in the traffic pattern flow, and X i It is the traffic pattern flow after decoupling and segmentation. It is the splicing result obtained by repeatedly looping each traffic mode flow.

[0048] Furthermore, the gated recurrent unit (GRU) in the time series module selectively stores and updates the hidden state of the input concatenated features through its gating mechanism, as expressed by the formula:

[0049]

[0050]

[0051]

[0052]

[0053] in, It is the traffic flow input at the current time step, updating gate z. t and reset door r t By inputting traffic flow at the current time step Hidden state H at the previous time step (t-1) The linear transformation and activation function are used to obtain the candidate hidden state at the current time step; after the reset gate controls the inflow of historical information, the linear transformation and activation function are used to obtain the candidate hidden state at the current time step. For H (t-1) and After updating, the output H of the hidden state at the current time step is obtained. (t) ∈R N×d ;

[0054] Finally, T h The outputs of time steps are stacked together to form an output tensor.

[0055]

[0056] Furthermore, the formula for expressing the predicted state result diagram is as follows:

[0057] H f =[(ReLU(ReLU(H)W1)W2)*W] T

[0058] Where H represents the multi-scale features obtained from skip connections, W1 and W2 are the parameter matrices of the fully connected layers, and W is the weight of the convolutional layer.

[0059] Furthermore, the dynamic graph generation module, decoupling module, residual graph convolution module, time series module, and skip connection layer module constitute a prediction model for spatiotemporal fusion graph generation. The prediction model is trained using the Adam optimization algorithm, and the loss is calculated using the MAE function to optimize the model. The applied system software uses a learning rate scheduler to automatically adjust the learning rate, with a decay rate of 0.5. A warm-up strategy is used in the first 20 training cycles, followed by course learning, where the course learning length increases every three cycles until it reaches 12. An early stopping mechanism is used to prevent overfitting, and the patience value is set to 100.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] 1. A novel method for generating adaptive spatiotemporal fusion graphs is proposed:

[0062] This method utilizes a similarity adjacency matrix generated by temporal embedding and a spatial graph generated by spatial information embedding, combined with an attention mechanism to dynamically adjust attention levels, thus forming a highly adaptive spatiotemporal graph. This method can not only assign unique dynamic spatiotemporal graphs to different traffic modes, but also comprehensively consider the correlation between nodes and the dynamic changes in traffic flow between different traffic modes.

[0063] 2. A novel traffic mode decoupling module is proposed:

[0064] By utilizing spatiotemporal features obtained through temporal and spatial embedding, complex traffic flows are decoupled into different traffic mode representations. This method effectively addresses the complex issues of interconnectivity and spatiotemporal heterogeneity in multimodal transportation systems. Extensive ablation experiments demonstrate that the decoupling module of this invention has a substantial positive impact on traffic flow prediction.

[0065] 3. A novel traffic flow prediction framework is proposed:

[0066] It can effectively represent the dynamic correlation between traffic patterns and adaptive spatiotemporal graphs, and extensive experiments were conducted on four large-scale real-world datasets to gain insights into the framework and instantiate its effectiveness. Experimental results consistently demonstrate significant advantages over all baseline methods. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the overall model architecture of a spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism, according to an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the feature extraction process of the residual map convolution module of the present invention;

[0069] Figure 3 A schematic diagram illustrating the process of feature extraction and stacking of the time series module of the present invention, followed by input into fully connected and two-dimensional convolutional layers;

[0070] Figure 4 This is a schematic diagram illustrating the information jump connection of the jump connection layer module of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0072] like Figure 1As shown in the figure, this invention proposes an attention-based spatiotemporal fusion graph generation method for multimodal traffic, which proposes an innovative traffic flow prediction network, SFADNet, to capture the spatiotemporal relationships of different traffic modes. It constructs a traffic flow decoupled spatiotemporal fusion graph, decoupling and segmenting complex traffic flows into different traffic mode flows through node spatial and temporal embeddings. For each mode, a different attention-based adaptive spatiotemporal fusion graph is assigned to each traffic mode flow for gated graph convolution, which helps capture the dynamic implicit relationships between time and space. The adaptive spatiotemporal fusion graph integrates temporal and spatial information graphs and their correlations, and uses residual graph convolution to better capture the dynamic spatiotemporal relationships under different fine-grained traffic modes. Finally, the outputs of multiple components are concatenated along the time dimension, with an RNN capturing short-term temporal information, and then concatenated with temporal periodic information from the temporal embeddings to capture multi-scale features, thereby accurately predicting traffic flow. The specific steps of this method are as follows:

[0073] Step 1: Build the dynamic graph generation module (Differentiable Graph-Generator, DGG).

[0074] In addressing the problem of modeling the spatiotemporal relationships of traffic patterns, this invention introduces an adaptive dynamic graph generation module, the goal of which is to learn the graph adjacency matrix to capture the potential relationships between spatiotemporal data.

[0075] Constructing a spatial graph involves two key issues: first, how to efficiently generate the graph with limited computational and memory resources; and second, how to accurately extract unidirectional relationships of node changes. Therefore, a sampling method is adopted, calculating pairwise relationships only between subsets of nodes, as shown in the following formula:

[0076] M1 = tanh(αW1E1)

[0077] M2=tanh(αW2E2)

[0078]

[0079]

[0080] Where E1∈R N×d ,E2∈R N×d W1 and W2 represent the randomly initialized node space embedding matrix, W1 and W2 are the parameter matrices of the fully connected layer, and α is a parameter that controls the saturation rate.

[0081] The pairwise matrices M1 and M2 are regularized using an activation function and subtraction terms to generate a spatial graph adjacency matrix A. s ∈R N×N This means obtaining the spatial graph. Then the matrix... The largest K elements in each row are retained, and the other elements are set to zero to obtain the adjacency matrix. The goal of this step is to retain only the top-K nearest node connections, thus sparsifying the adjacency matrix to reduce computational complexity.

[0082] The key step in constructing a time graph lies in how to use time information to build a symmetric matrix representing the temporal correlation of nodes, as shown in the formula below:

[0083]

[0084]

[0085] in, Indicates the total traffic flow X in The time period matrix obtained in the first step is to reduce the dimension T. h Average pooling, for the first T h Taking the average of the step sizes reduces the spatial dimension, resulting in matrix T. D ∈R N×d ,T W ∈R N×d .

[0086] m1 = tanh(T) D (T D ) T )

[0087] m2=tanh(T W (T W ) T )

[0088]

[0089]

[0090] By analyzing matrix T D and T W Performing the transpose dot product operation yields the similarity matrix m1∈R. N×N m2∈R N×N Then, the time-plot relationship matrix A is generated through subtraction terms. t ∈R N×N That is, obtain the time map, and then perform a masking operation to retain the top-k nearest node connections.

[0091] Then, for the spatial graph adjacency matrix A s Relationship matrix A with time graph t A multi-head attention mechanism is used to assign weights to better capture the correlation between the spatial and temporal graphs during the fusion process, as shown in the following formula:

[0092] A f =ReLU(tanh(βA) s A t ))

[0093]

[0094] Multihead s (A s A t A f ) = (head1||head2||…||head s W o

[0095]

[0096] Where β is a learnable parameter, A f ∈R N×N Spatial graph A was further captured by using activation functions and scaling. s And time graph A t The interaction effect. The query variable is spatial graph A. s ∈R N×N and learnable query matrix Multiplication, the key variable is time plot A. t ∈R N×N and learnable key matrix Multiply the query vector and the key vector to calculate their similarity, then divide the similarity by 1 / 2. Scaling is then applied. The scale similarity is further processed using the softmax function to obtain the attention weights W. o Finally, attention weight values ​​are used to adjust the key value variables. Weighting, The learnable value matrix enables the output spatiotemporal fusion graph to dynamically allocate weights by extracting the similarity relationships between spatiotemporal graphs obtained from different traffic modes, thereby improving the ability to model spatiotemporal relationships.

[0097] Step 2: Construct the decoupling module.

[0098] To extract the heterogeneous spatiotemporal features of different traffic modes, a decoupling module is introduced to segment the traffic modes. This invention starts from the input total traffic flow. The periodic features of time and date are extracted, and then embedded using two randomly initialized learnable time slot embedding matrices. This is transformed into a continuous real-valued vector representation, i.e., a time period matrix. and Where, N dThis represents the number of time slots within a day, specifically determined by the sensor's sampling frequency, N. W The feature dimension of the time matrix represents the number of days in a week, and its size is determined by E. D and E W The embedding dimension determines this.

[0099] Then, the node space embedding matrix E is initialized using a learnable parameter. i ∈R N×d This method captures the semantic information and feature representation of each node in the traffic flow network. Nonlinear transformations and feature extraction are performed on the input features using fully connected layers and the modified ReLU linear activation function. To estimate the proportional relationship between two hidden traffic flows, a decoupling module value between 0 and 1 is generated using the Sigmoid activation function. Then, the decoupling module value is multiplied element-wise with the original traffic flow data to achieve weighted representations of different traffic modes, resulting in different traffic mode flows. This process effectively decouples the traffic flow data, facilitating the segmentation of different traffic modes.

[0100] The decoupling module used to separate traffic flow into two traffic modes can be expressed as:

[0101]

[0102] X1 = X in ⊙Ω

[0103] X2 = X in -X1

[0104] Wherein, the weight matrix W1∈R 3d×d W2∈R d×1 A linear transformation is performed on the feature vector, and nonlinear activation functions ReLU and Sigmoid are introduced to obtain Ω. t,i ∈(0,1), representing the proportion of a specific traffic pattern at node i relative to the total traffic flow at time step t. Matrix Represents a specific traffic mode relative to the total traffic flow in T h The output, representing the proportion at each time step and N nodes, is broadcast via Hadamard product to... All channels. Decoupled from the total traffic flow X by the proportional matrix Ω. in Perform element-wise multiplication to obtain a partial traffic flow. This indicates the contribution of a specific transportation mode. Subsequently, through X... in Subtracting the aforementioned traffic flow X1 from the traffic flow yields another traffic flow X2, which contains flow not explained by a specific traffic pattern.

[0105] Step 3: Construct the Residual Graph Convolutional (RGC) module.

[0106] The residual graph convolution module aims to perform graph convolution on the adaptive spatiotemporal fusion graph generated by the dynamic graph generation module for each traffic mode flow, utilizing the relationships between nodes and edges in the graph structure to capture the spatial and temporal dependencies in the traffic network. Adjacency matrix A and its transpose A T As input, the outputs of the two residual graph convolutional layers are combined by addition to obtain the fused graph information.

[0107] The formula for calculating the residual plot convolutional layer is as follows:

[0108]

[0109]

[0110] h out =(h (0) ||h (1) ||…||h (k-1) W

[0111] Where A represents the adjacency matrix. It is an adjacency matrix with added self-joins, and a degree matrix. Meet the conditions k represents the propagation depth, h in and h out These represent the outputs of the previous residual map convolutional layer and the current residual map convolutional layer, respectively. W is the convolutional layer weight, and the parameter α indicates the number of input nodes h to retain during node updates. in The goal of this step is to find a balance in the update of node representations to ensure that local features are preserved and depth exploration is performed simultaneously, such as... Figure 2 As shown.

[0112] Step 4: Construct the Time Series (TS) module.

[0113] The spatiotemporal features extracted from the decoupled traffic flow data after passing through the residual graph convolution module The splicing formula is as follows:

[0114]

[0115]

[0116] Where N is the number of iterations in the residual graph convolutional module, P is the number of decouplings in the traffic pattern flow, and X i It is the concatenated result of multiple iterations of each traffic mode flow after decoupling and segmentation. Input time series module.

[0117] The gated cyclic unit (GRU) in the time series module allows for the selective storage and updating of hidden states through its gating mechanism, which helps to capture periodic patterns and dependencies in the input traffic flow data.

[0118]

[0119]

[0120]

[0121]

[0122] in, It is the traffic flow input at each time step, updating gate z. t and reset door r t By inputting traffic flow at the current time step Hidden state H at the previous time step (t-1) The linear transformation and activation function are used to obtain the candidate hidden state at the current time step. This is achieved by controlling the inflow of historical information through a reset gate, and then by the linear transformation and activation function. For H (t-1) and After updating, the output H of the hidden state at the current time step is obtained. (t) ∈R N×d and T h The outputs of the time steps are stacked together, such as Figure 3 As shown, the output tensor is formed.

[0123]

[0124] Step 5: Design output and training strategies.

[0125] like Figure 4 As shown, important information flows within the model are connected in a skip-connection manner:

[0126] H = {H out ||X out ||X in ||T D ||T W}

[0127] obtained by jump connection Then, a regression layer is formed using fully connected layers and two-dimensional convolutions to generate the predicted state.

[0128] H f=[(ReLU(ReLU(H)W1)W2)*W] T

[0129] Given the true value The formula for optimizing the model by calculating the loss using the MAE function is as follows:

[0130]

[0131] Step Six: Model Training.

[0132] The experiments were conducted on a system equipped with an NVIDIA GeForce RTX 4090 GPU. Datasets PEMS03, PEMS04, PEMS07, and PEMS08 were divided into training, validation, and test sets in a 6:2:2 ratio. The model was trained using the Adam optimization algorithm with a weight decay (wdecay) of 1.0e-5 and an epsilon (eps) of 1.0e-8. Furthermore, a learning rate scheduler was used to automatically adjust the learning rate, with a decay rate of 0.5. A warm-up strategy was employed for the first 20 training epochs, followed by course learning, where the course length increased every three epochs until it reached 12. An early stopping mechanism was used to prevent overfitting, with a patience value set to 100. The configurations of the model's relevant parameters on the four datasets are shown in Table 2, including the node embedding dimension N, the temporal embedding dimension D, the top-k dimensions k_s and k_t of the spatial and temporal adjacency matrices, and the number of decoupled traffic patterns G.

[0133] Three commonly used metrics in traffic forecasting were used to evaluate the performance of all baseline models: (1) MAE (mean absolute error), (2) root mean square error (RMSE), and (3) mean absolute percentage error (MAPE), as shown in the following formulas:

[0134]

[0135]

[0136]

[0137] Among them, MAE reflects prediction accuracy, RMSE is sensitive to larger errors, and MAPE is more concerned with relative errors. i Let Ω represent the i-th predicted value, and let Ω represent the length of the predicted sequence. In the experiment, Ω = 12.

[0138] Step 7: Experimental Results.

[0139] The table below compares the mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) for all methods across four datasets:

[0140]

[0141] In summary, the following conclusions can be drawn:

[0142] (1) Non-graphical models such as FC-LSTM and DSANet only use deep learning methods to capture temporal features from traffic data, ignoring the spatial correlation in traffic networks, and thus have poor performance.

[0143] (2) Graph-based algorithms such as GraphWaveNet perform superiorly, demonstrating the effectiveness of graphs. However, DCRNN, by capturing spatial features using a predefined graph, is limited by the representation of inter-node correlations. In contrast, STGODE and STG-NCD proposed ODE and NCDE to increase their spatial receptive domains. However, they failed to mine global spatial correlations, and therefore performed worse than DSTAGNN.

[0144] (3) In addition to the MAPE metric on the PEMS03 dataset, our SFADNet achieved the best results on other metrics of the four datasets, proving that the spatiotemporal fusion graph generated by our SFADNet using periodic features and spatial embedding has a good global spatiotemporal feature capture capability and performs well on different datasets.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Construct a dynamic graph generation module to obtain and construct temporal and spatial graphs from total traffic flow data. Then, based on an attention mechanism, fuse the temporal and spatial graphs and obtain the correlation between them to generate an adaptive spatiotemporal fusion graph. Step 2: Construct a decoupling module to decouple the total traffic flow into different traffic mode flows. Specifically, a node spatial embedding matrix initialized with learnable parameters is used to capture the semantic information and feature representation of each node in the traffic flow network, through fully connected layers and modified linear activation functions. Nonlinear transformation and feature extraction were performed on the features, using... The activation function generates a decoupling module value to estimate the proportional relationship between traffic mode flows; then, the decoupling module value is multiplied element-wise with the total traffic flow data to obtain different traffic mode flows. Step 3: Construct a residual graph convolution module to perform graph convolution on the generated adaptive spatiotemporal fusion graph of each traffic mode flow to extract the spatiotemporal features of each traffic mode flow; Step 4: Construct a time series module. The spatiotemporal features extracted from each traffic pattern flow are spliced ​​together and input into the time series module. The hidden state of the traffic pattern flow at its total time step is further obtained through a gating mechanism. Step 5: Construct a jump connection layer module to perform jump connections on the total traffic flow, time map, decoupled traffic pattern flow, and the hidden state of the total time step to obtain multi-scale features. Then, a regression layer is formed through fully connected layers and two-dimensional convolution to generate a traffic flow prediction state result map. The construction of the spatial graph is specifically expressed as follows: in This represents a randomly initialized node space embedding matrix. It is the parameter matrix of the fully connected layer. It is a parameter for controlling the saturation rate; Paired matrices , The spatial graph adjacency matrix is ​​generated by activation function regularization and subtraction terms, thus obtaining the spatial graph. Then the matrix The largest in each row Keep one element and set the other elements to zero to obtain the adjacency matrix. ; The specific expression for constructing the time graph is as follows: in, Indicates the total traffic flow The time period matrix obtained in the first step is to adjust the dimensions. Average pooling, for the first Taking the average of the step sizes reduces the spatial dimension, resulting in a matrix. ; By analyzing the matrix and Perform the transpose dot product operation to obtain the similarity matrix. Then, the time-map relationship matrix is ​​generated by subtraction, thus obtaining the time-map. Then proceed Masking operations are reserved Connect to the nearest node.

2. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The spatial graph adjacency matrix Relationship matrix between time and graph A multi-head attention mechanism was used to assign weights to capture the correlation between the spatial and temporal graphs during the fusion process. The formula is expressed as: in, It is a learnable parameter. The spatial map was further captured by using activation functions and scaling. and time graph Interaction effects; spatial graph and learnable query matrix Multiply to obtain the query variable, time graph and learnable key matrix Multiply to obtain the key variable, calculate the similarity between the query vector and the key vector, and then divide the similarity by . Scaling; via The function further processes the scaled similarity to obtain attention weights. Finally, attention weight values ​​are used to evaluate key value variables. Weighting, The learnable value matrix enables the output spatiotemporal fusion graph to dynamically allocate weights by extracting the similarity relationships between spatiotemporal graphs obtained from different traffic modes.

3. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The formula for the decoupling module used to separate the total traffic flow into two traffic modes is expressed as follows: Among them, the weight matrix , A linear transformation is performed on the feature vectors, and a nonlinear activation function is introduced. and get , representing the proportion of a specific traffic pattern at node i relative to the total traffic flow at time step t; matrix Represents a specific traffic mode relative to total traffic flow. Each time step and The proportion output at each node is broadcast to [the relevant node] via the Hadamard product. All channels; via decoupling scaling matrix With total traffic flow Perform element-wise multiplication to obtain a partial traffic flow. This indicates the contribution of a specific transportation mode; subsequently, by... Subtract the aforementioned traffic flow To obtain another traffic flow This includes traffic flow that is not explained by a specific traffic pattern.

4. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The formula for graph convolution performed by the residual graph convolution module in step three is as follows: in Represents the adjacency matrix. It has added a self-connection. Adjacency matrix, degree matrix And satisfy , Indicates the depth of dissemination. Indicates the weights of the convolutional layer. and These represent the outputs of the previous and current residual map convolutional layers in the residual map convolution module, respectively. This indicates that the input node is retained when the node is updated. The proportion.

5. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The formula for spatiotemporal feature splicing in step four is expressed as follows: Where N is the residual map convolution module The number of iterations, P is the number of decouplings in the traffic pattern flow. It is the traffic pattern flow after decoupling and segmentation. The stitched result is obtained by repeatedly looping each traffic mode flow.

6. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 5, characterized in that, The gated recurrent unit (GRU) in the time series module selectively stores and updates the hidden state of the input concatenated features through its gating mechanism, as expressed by the formula: in, It is the traffic flow input at the current time step, updating the gate. and reset door By inputting traffic flow at the current time step Hidden state of the previous time step The linear transformation and activation function are used to obtain the candidate hidden state at the current time step; after the reset gate controls the inflow of historical information, the linear transformation and activation function are used to obtain the candidate hidden state at the current time step. ,right and After updating, the output of the hidden state at the current time step is obtained. ; Finally The outputs of time steps are stacked together to form an output tensor. : 。 7. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The formula for expressing the predicted state result diagram is: in, The multi-scale features obtained from skip connections It is the parameter matrix of the fully connected layer. These are the weights of the convolutional layer.

8. The spatiotemporal fusion graph generation method for multimodal traffic based on an attention mechanism as described in claim 1, characterized in that, The dynamic graph generation module, decoupling module, residual graph convolution module, time series module, and skip connection layer module constitute a prediction model for spatiotemporal fusion graph generation. The prediction model is trained using the Adam optimization algorithm, and the loss is calculated using the MAE function to optimize the model. The applied system software uses a learning rate scheduler to automatically adjust the learning rate, with a decay rate of 0.

5. A warm-up strategy is used in the first 20 training cycles, followed by course learning, where the course learning length increases every three cycles until it reaches 12. An early stopping mechanism is used to prevent overfitting, and the patience value is set to 100.

Images