Traffic flow prediction device, prediction method and prediction model construction method

Through the MCB and Attention dual-channel multimodal fusion model, the spatial and temporal graph features, visual features and weather factors are integrated, which solves the problem of low accuracy of existing traffic flow prediction methods and achieves higher precision traffic flow prediction.

CN115985104BActive Publication Date: 2024-11-19长三角信息智能创新研究院
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310136816.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-10
Publication Date
2024-11-19
Estimated Expiration
2043-02-10

AI Technical Summary

Technical Problem

Existing traffic flow prediction methods are susceptible to various factors, resulting in low prediction accuracy and inability to effectively deal with the problem of space-time dependence.

Method used

Using a dual-channel multimodal fusion model based on MCB and Attention, the space-time graph characteristics of traffic flow, road network trajectory visual characteristics and weather factor vectors are fused to build a traffic flow prediction model, and the spatial-time graph characteristics are extracted through sliding window and weight fusion. and consider the impact of time decay of weather factors.

Benefits of technology

The accuracy of traffic flow prediction is improved, the impact of other factors on the prediction results is reduced, and a higher-precision traffic flow prediction model is constructed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115985104B_ABST
    Figure CN115985104B_ABST
Patent Text Reader

Abstract

The present invention discloses a traffic flow prediction device, a prediction method and a prediction model construction method, and belongs to the technical field of traffic flow prediction. Based on the traditional time series prediction model, the present invention constructs a traffic flow spatiotemporal graph based on the time series data of traffic sensors, and uses the weighted-STSGCN algorithm to extract the graph structure information of the traffic flow spatiotemporal graph to obtain the spatiotemporal graph features; generates road network and trajectory images based on road network and trajectory data, and uses ResNet to extract image features to obtain the visual features of the road network and trajectory; finally, the spatiotemporal graph features and visual features are fused by a dual-channel multimodal fusion model based on MCB and Attention, and combined with weather factors, so as to construct a prediction model for realizing future traffic flow. The model constructed by the present invention can effectively achieve the prediction accuracy of traffic flow.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic flow prediction, and more specifically, relates to a traffic flow prediction device, a prediction method and a prediction model construction method based on MCB and Attention dual-channel multimodal fusion. Background Art

[0002] Transportation is the most important infrastructure in modern cities, providing daily travel and mobility services for millions of people. With the acceleration of urbanization, the continuous increase in population, the improvement of road traffic, and the increasing complexity of the transportation system, it includes road traffic, rail transit, pedestrians, and a variety of shared transportation tools such as online car-hailing and shared bicycles. However, with the acceleration of urbanization, many corresponding problems have also emerged in the development of cities, such as air pollution and traffic congestion. Early intervention based on traffic prediction is considered to be an important method to improve the efficiency of the transportation system and alleviate traffic congestion. With the development of intelligent cities and intelligent transportation systems, sensors such as ring detectors are set up on the road to sense the traffic status. The on-board GPS system can continuously read the vehicle location information to feedback the traffic status, transaction records of subways, buses, etc., and road condition monitoring videos. Even smart phones equipped with GPS can be used to collect data on pedestrian travel and indirectly reflect road condition information. The main process of traffic prediction is to predict future traffic status based on the above-mentioned historical traffic-related data and combined with environmental factors (weather, holidays, POI, etc.).

[0003] Traffic forecasting is very challenging due to the large amount of data, high dimension, strong dynamics, and various emergencies such as traffic accidents. Traffic condition prediction has both spatial and temporal dependencies, and will be affected by nearby areas and seasonal time. Traditional traffic forecasting models mainly use linear time series models, such as the autoregressive integrated moving average model (ARIMA), which cannot handle such spatiotemporal dependency problems. In order to solve the shortcomings of linear time series models, related studies have introduced data-driven machine learning and deep learning technologies. However, although existing methods can capture the spatiotemporal dependency problems of traffic forecasting problems, traffic forecasting is still inevitably affected by various factors. Summary of the invention

[0004] 1. Problem to be solved

[0005] One of the purposes of the present invention is to provide a traffic flow prediction device and prediction method based on MCB and Attention dual-channel multimodal fusion, so as to solve the problem that the existing prediction method is easily affected by various factors and improve the prediction accuracy of traffic flow.

[0006] A second object of the present invention is to provide a method for constructing a traffic flow prediction model, so that multimodal fusion of various factors affecting traffic flow prediction can be performed to obtain a traffic flow prediction model with higher prediction accuracy.

[0007] 2. Technical solution

[0008] In order to solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] A method for constructing a traffic flow prediction model of the present invention comprises the following steps:

[0010] Step 1: construct a spatiotemporal graph of traffic flow, extract the spatiotemporal graph features of traffic flow, and generate a spatiotemporal graph representation vector;

[0011] Step 2: construct an image sequence based on the road network, vehicle trajectory and sensor position data, extract features from the image sequence, and obtain a visual representation vector of the image sequence;

[0012] Step 3: Extract the characteristics of weather factors to form a vector representation of weather factors;

[0013] Step 4: Use the MCB and Attention dual-channel multimodal fusion model to fuse the spatiotemporal graph representation vector with the visual representation vector of the image sequence to obtain a fusion vector that aligns the graph structure with the visual features, and finally splice the weather factor vector to obtain a traffic flow prediction model.

[0014] In view of the shortcomings of existing traffic flow prediction methods that are easily affected by various factors, resulting in low prediction accuracy, the present invention performs multimodal fusion of various factors affecting traffic flow prediction, that is, fusing and splicing the spatiotemporal graph representation vector of traffic flow, the visual representation vector of picture sequence and the weather factor vector, thereby effectively improving the prediction accuracy of the traffic flow prediction model.

[0015] Furthermore, when constructing the spatiotemporal graph of traffic flow in step 1, spatial graphs at n different moments are constructed respectively according to the relevant data of the traffic flow sensor; for the spatial graph at n moments, all nodes are connected to themselves at adjacent time steps as the time edges between sensor nodes, thereby obtaining the global spatiotemporal graph of the sensor for n moments; the spatial edge weights of the sensor nodes The calculation method is shown in formulas (1) and (2):

[0016]

[0017]

[0018] In the above formula, d α,βis the spatial distance between sensor α and sensor β, is the spatial score between sensor α and sensor β;

[0019] Temporal edge weights of sensor nodes The calculation method is shown in formulas (3) and (4):

[0020]

[0021]

[0022] In the above formula, N is the number of sensors, is the average speed of vehicles in the area, △t is the time interval between two moments, is the temporal edge score of the sensor node.

[0023] Furthermore, in step 1, an improved STSGCN is used to extract features from the spatial graph structure, which specifically includes spatiotemporal graph serialization based on sliding windows and local feature extraction of STSGCM based on weight fusion, where:

[0024] (1) Spatiotemporal graph serialization based on sliding windows

[0025] The global space-time graph is serially segmented by using a sliding window combined with sequence filling to obtain a series of local space-time graphs.

[0026] (2) STSGCM local feature extraction based on weight fusion

[0027] Based on STSGCM, the segmented spatiotemporal graph sequences are respectively feature extracted to obtain the corresponding spatiotemporal graph feature sequences; then the features of each spatiotemporal graph sequence are fused based on the weights to finally obtain the output vector sequence, i.e., the spatiotemporal graph representation vector.

[0028] Furthermore, when serializing and segmenting the global space-time graph, a sliding window with a window size of 3 and a step size of 1 is used to serialize and segment the time direction, and t is added to the constructed global space-time graph. 0 and t n+1 The spatial graph is randomly initialized at every moment to reduce the feature loss of feature extraction; after serialization segmentation, a local spatiotemporal graph with n time steps of 3 is obtained.

[0029] Furthermore, when the feature extraction of the segmented spatiotemporal graph sequence is performed based on STSGCM, the specific formula of STSGCM operation is shown in formula (6) and formula (7):

[0030] H (l) =σ(A′H (l-1) W+b) (6)

[0031]

[0032] Where A′ represents the adjacency matrix of the local spatiotemporal graph with a time step of 3, and H (l-1) is the input of the lth graph convolutional layer, W and b are learnable parameters, σ is the activation function, represents the j-th layer graph convolution output vector of the i-th local empty graph sequence, λ 1 +λ 2 =1, L is the number of graph convolution layers, It is the graph convolution fusion output result of the i-th local empty graph sequence.

[0033] Furthermore, the specific operation of fusing the features of the spatiotemporal graph sequence based on the weights is shown in the following formula:

[0034]

[0035] in, Represents the j-th layer graph convolution output vector at three moments of the i-th local spatiotemporal graph sequence, ω i-1 +ω i +ω i+1 =1, and substituting it into formula (7) we get:

[0036]

[0037] After feature extraction of all local spatiotemporal graph sequences, the output vector sequence is finally obtained

[0038] Furthermore, in step 3, a weather factor extraction algorithm based on time decay is used to extract weather-related factors. The specific process is as follows:

[0039] (1) In the model training stage, the time series feature vectors of the four weather factors are constructed based on the historical meteorological data of rainfall, temperature, air pressure, and wind speed as the model input data;

[0040] (2) In the model prediction stage, weather forecast data for a certain time interval in the future are selected as initial data, and a polynomial-based method is used to correct the weather forecast data as the weather factor input data for model prediction. The specific calculation process is shown in formula (11):

[0041]

[0042] in, c=1,2,…,t-1 represent the actual rainfall, temperature, air pressure and wind speed at time c respectively. They represent the rainfall, temperature, air pressure, and wind speed of the weather forecast at time t, respectively. Respectively represent the rainfall, temperature, air pressure and wind speed of the revised weather forecast at time t, μ c (c=1,2,…,t-1) is the historical weather correction parameter, η is the current weather forecast correction parameter, μ c and η are calculated based on the mean of historical statistical results.

[0043] Furthermore, the specific process of step 4 is as follows:

[0044] (1) The MCB algorithm is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain the fusion result vector

[0045] (2) A cross-modal attention mechanism is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain a fusion vector based on the attention mechanism.

[0046] (3) and the weather factor vector V weather The splicing is performed and the traffic flow is predicted to obtain the traffic flow prediction result Y at the next moment. The specific calculation process is shown in formulas (15)-(17):

[0047]

[0048]

[0049]

[0050] Among them, W 1 ,b 1 ,W 2 ,b 2 are all learnable parameters, γ 1 +γ 2 =1, Represents the fusion vector of spatiotemporal graph features and visual features, A fusion vector representing spatiotemporal features, visual features, and weather features.

[0051] A traffic flow prediction method of the present invention comprises the following steps:

[0052] Step 1: Get data

[0053] Obtain relevant data from traffic flow sensors, road network and vehicle GPS trajectory information, and weather and environmental factor information;

[0054] Step 2: Based on the data obtained in step 1, a traffic flow prediction model is constructed according to the method described in any one of claims 1 to 7;

[0055] Step 3: Predict the traffic flow at the next moment based on the constructed prediction model.

[0056] The traffic flow prediction device of the present invention comprises:

[0057] A space-time graph representation vector construction module is used to construct a space-time graph of traffic flow based on relevant data of traffic flow sensors, extract the space-time graph features of traffic flow, and generate a space-time graph representation vector;

[0058] A visual representation vector construction module for image sequences is used to generate road network and trajectory image sequences at different times based on road network data, and to extract features from the generated road network and trajectory image sequences to obtain visual feature representation vector sequences;

[0059] A vectorized representation building module for weather factors, which is used to extract the features of weather factors and form a vectorized representation of weather factors;

[0060] And a fusion module is used to fuse the generated spatiotemporal graph representation vector, visual feature representation vector sequence and vectorized representation of weather factors based on the dual-channel fusion model of MCB and Attention to form a traffic flow prediction model and predict the traffic flow at the next moment.

[0061] 3. Beneficial effects

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) A traffic flow prediction model construction method of the present invention is based on a multimodal fusion model of MCB and Attention, which fuses the graph structure features of the spatiotemporal graph and the visual features of the road network trajectory, and splices the weather characteristics with the fusion features, thereby constructing a traffic flow prediction model with higher prediction accuracy and minimizing the influence of other factors on the prediction results.

[0064] (2) A traffic flow prediction model construction method of the present invention, based on the traditional time series prediction model, is based on the time series data of traffic sensors and uses the time-space matrix conversion method of vehicle speed to construct a traffic flow spatiotemporal graph, and improves the STSGCN to form a weighted-STSGCN model. By using padding and weight fusion methods, the structural characteristics of the traffic flow spatiotemporal graph can be efficiently extracted, and the node feature information of the fusion result can be made more comprehensive.

[0065] (3) A method for constructing a traffic flow prediction model of the present invention associates the road network and vehicle GPS trajectory information, generates road network and trajectory images, and uses ResNet to extract image features, which can effectively convert the road network and trajectory information into visual features, thereby helping to improve the effect of the prediction model; at the same time, the present invention also takes into account the influence of weather and environmental factors, extracts weather factor features, and forms a vectorized representation of weather factors, which is conducive to further improving the effect of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is an overall flow chart of the traffic flow prediction method of the present invention;

[0067] Figure 2 A flow chart for extracting spatiotemporal features of traffic flow according to the present invention;

[0068] Figure 3 A flowchart of visual feature extraction of a road network and a trajectory of the present invention;

[0069] Figure 4 It is the overall structure diagram of the traffic flow prediction model of the present invention. DETAILED DESCRIPTION

[0070] The key points and protection points of the present invention are: (1) a spatiotemporal matrix transformation method based on vehicle speed is designed to construct a weighted spatiotemporal graph of traffic flow, and the STSGCN graph feature extraction model is improved to obtain a weighted-STSGCN model; (2) an image sequence generation method for the fusion of road network and vehicle trajectory is proposed, which can effectively convert the road network and trajectory into visual features; (3) an MCB and Attention dual-channel multimodal fusion model is proposed to fuse spatiotemporal graph features and visual features, and weather features are integrated to achieve high-precision traffic flow prediction.

[0071] The present invention is further described below in conjunction with specific embodiments.

[0072] Example

[0073] Combination Figure 1 In this embodiment, a dual-channel multimodal fusion model based on MCB and Attention is used to fuse the spatiotemporal graph features of traffic flow and the visual features of the trajectory road network, and weather and environmental factors are combined to realize the construction of a future traffic flow prediction model and the prediction of traffic flow. The specific steps are as follows:

[0074] Step 1: Extracting spatiotemporal features of traffic flow

[0075] The spatiotemporal graph feature extraction of traffic flow includes three parts: spatiotemporal graph construction of traffic flow, spatiotemporal graph encoding of traffic flow, and spatiotemporal graph feature extraction based on GCN (Graph Convolutional Network). Figure 2 As shown, it mainly includes the following steps:

[0076] 1. Construction of spatiotemporal graph of traffic flow

[0077] The traffic flow spatiotemporal graph is constructed based on the relevant data of the traffic flow sensor. 1 All sensors are regarded as nodes at all times, and the connections between sensors are used as spatial edges to construct t 1 The spatial diagram at time t 2 , t 3 …, t n A corresponding spatial graph is also constructed at each moment to obtain a spatial graph of n moments. For the spatial graph of n moments, all nodes are connected to themselves at adjacent time steps as the time edges between sensor nodes, so that the global spatiotemporal graph of the sensor corresponding to n moments can be obtained.

[0078] Spatial edge weights of sensor nodes The calculation method is shown in formula (1)-(2):

[0079]

[0080]

[0081] Among them, d α,β is the spatial distance between sensor α and sensor β.

[0082] Temporal edge weights of sensor nodes The calculation method is shown in formula (3)-(4):

[0083]

[0084]

[0085] Where N is the number of sensors, is the average speed of vehicles in the area (which can be estimated based on sampled data), △t is the time interval between two moments, is the temporal edge score of the sensor node.

[0086] 2. Spatial-temporal graph coding of traffic flow

[0087] Adding spatiotemporal embedding to the spatiotemporal network sequence allows the model to consider spatiotemporal information at the same time, which can enhance spatiotemporal correlation. g ∈RN×C×T , construct a learnable time embedding matrix and a learnable space embedding matrix, so that after the model training is completed, the two embedding matrices include time information and space information, as shown in formula (5):

[0088]

[0089] Among them, T e and S e They are the time embedding matrix and the spatial embedding matrix respectively, ⊕ represents the broadcast operation, C represents the sensor node characteristics (longitude, latitude, traffic flow, etc.), and T represents the time series length.

[0090] 3. Spatiotemporal graph feature extraction based on weighted-STSGCN

[0091] In order to extract the characteristics of the spatiotemporal graph of traffic flow, this embodiment uses an improved STSGCN (called weighted-STSGCN) to extract the characteristics of the spatial graph structure. The whole mainly includes two parts: spatiotemporal graph serialization based on sliding windows and STSGCM local feature extraction based on weight fusion:

[0092] (1) Spatiotemporal graph serialization based on sliding windows

[0093] Based on the model structure of STSGCN, it is necessary to serialize and segment the global spatiotemporal graph of traffic flow. This embodiment uses a sliding window combined with sequence padding to serialize and segment the global spatiotemporal graph to obtain a series of local spatiotemporal graphs. Specifically, a sliding window with a window size of 3 and a step size of 1 is used to serialize and segment the time direction, and t is added to the constructed global spatiotemporal graph. 0 and t n+1 The spatial graph is randomly initialized at each time to reduce the feature loss of feature extraction. After serialization segmentation, we will get n local spatiotemporal graphs with a time step of 3.

[0094] (2) STSGCM local feature extraction based on weight fusion

[0095] The local feature extraction of STSGCM based on weight fusion mainly includes two parts: local spatiotemporal graph feature extraction based on STSGCM and feature fusion based on weight. The specific process is as follows:

[0096] a) Local spatiotemporal graph feature extraction based on STSGCM

[0097] STSGCM consists of a set of graph convolution (GCN) operation modules. Each GCN module extracts features from the segmented spatiotemporal graph sequence to obtain the corresponding spatiotemporal graph feature sequence. The GCN operation can aggregate the features of each node with its adjacent nodes. Its input is the graph adjacency matrix of the local spatiotemporal graph, so that each node can aggregate its own and adjacent node features at adjacent time steps. The specific formulas of STSGCM operation are shown in (6)-(7):

[0098] H (l) =σ(A′H (l-1) W+b) (6)

[0099]

[0100] Where A′ represents the adjacency matrix of the local spatiotemporal graph with a time step of 3, and H (l-1) is the input of the lth graph convolutional layer, W and b are learnable parameters, σ is the activation function (such as ReLU), represents the j-th layer graph convolution output vector of the i-th local empty graph sequence, λ 1 +λ 2 =1, L is the number of graph convolution layers.

[0101] b) Weight-based feature fusion

[0102] Based on the weight fusion process, all the features of the nodes in the previous time step and the next time step can be used, so that the node feature information of the fusion result is more comprehensive, while including the local spatiotemporal correlation and eliminating the feature redundant information. The specific operation process is shown in formula (8):

[0103]

[0104] in, Represents the j-th layer graph convolution output vector at three moments of the i-th local spatiotemporal graph sequence, ω i-1 +ω i +ω i+1 =1, and substituting it into formula (7) we get:

[0105]

[0106] After feature extraction of all local spatiotemporal graph sequences, the output vector sequence is finally obtained i=1,2,...,n.

[0107] Step 2: Visual feature extraction of road network and trajectory

[0108] In order to improve the performance of the prediction model, this embodiment associates the road network and vehicle GPS trajectory information to construct visual features to improve the model effect, which specifically includes the generation of road network and trajectory image sequences and the extraction of visual features based on the road network and trajectory image sequences. The overall process is as follows: Figure 3 As shown, the main steps are as follows:

[0109] 1. Road network and trajectory image sequence generation

[0110] Based on the road network data (including expressways, urban expressways, urban main roads, urban secondary roads, urban branch roads, rural roads, bicycle lanes, pedestrian roads, internal roads, and other 10 types of road data), the road network data is visualized using relevant software (such as ArcGIS) to obtain a road network visualization map, and the sensor locations are marked on the visualized road network map. According to the longitude and latitude data with timestamps such as taxis, buses, bicycles, and pedestrians, the number of people on the road network between any two sensors in the time interval T is counted. k The number of vehicles and pedestrians at the time and the proportion are calculated. The calculation formula is shown in (10):

[0111]

[0112] in, T represents the number of taxis, buses, bicycles, and pedestrians between sensor i and sensor j. k The number of moments, Texi k ,Bus k ,Bicycle k ,Human k They represent taxis, buses, bicycles, and pedestrians in T k The total number of moments.

[0113] The calculated proportion is used as the influence index, and road network trajectories of different color depths are constructed between sensors according to the size of the influence index. The results are annotated on the visual road network map, and finally a sequence of road network and trajectory images at different times is generated.

[0114] 2. Visual feature extraction based on road network and trajectory image sequences

[0115] For the generated road network and trajectory image sequences ResNet is used for feature extraction to obtain a sequence of visual feature representation vectors

[0116] Step 3: Quantization of weather factor features

[0117] In order to further improve the performance of the prediction model, after obtaining the graph structure features and visual features, this embodiment takes into account the influence of weather environmental factors (rainfall, temperature, air pressure, wind speed, etc.), extracts the weather factor features, and forms a vectorized representation of the weather factors. In the prediction stage, since the future weather is unknown, the model can only use relevant weather forecast data for prediction, which will cause the accuracy of the weather forecast to affect the accuracy of the prediction model. At the same time, the accuracy of the weather forecast will also decay over time, so this embodiment uses a weather factor extraction algorithm based on time decay to extract weather-related factors. The specific process of weather factor extraction is as follows:

[0118] During the model training phase, the time series feature vectors of four weather factors are constructed as model input data based on the historical meteorological data of rainfall, temperature, air pressure, and wind speed.

[0119] In the model prediction stage, weather forecast data (rainfall, temperature, air pressure, wind speed) at a certain time interval in the future are selected as initial data, and a polynomial-based method is used to correct the weather forecast data as the weather factor input data for model prediction. The specific calculation process is shown in formula (11):

[0120]

[0121] in, c=1,2,…,t-1 represent the actual rainfall, temperature, air pressure and wind speed at time c respectively. They represent the rainfall, temperature, air pressure, and wind speed of the weather forecast at time t, respectively. Respectively represent the rainfall, temperature, air pressure and wind speed of the revised weather forecast at time t, μ c , c=1,2,…,t-1 is the historical weather correction parameter, η is the current weather forecast correction parameter, μ c and η are calculated based on the mean of historical statistical results.

[0122] Step 4: Multimodal fusion traffic flow prediction model

[0123] After completing the three modal feature extractions in steps 1, 2, and 3, a dual-channel fusion model based on MCB and Attention is used to fuse the multimodal features of graph structure and vision, and combined with weather factor features to predict future traffic flow. The overall model structure is as follows: Figure 4 As shown, it is divided into the following steps:

[0124] 1. Multimodal fusion based on MCB

[0125] For the feature vector sequence of the space-time graph i=1,2,...,n and the visual vector sequence of the road network and trajectory The MCB algorithm is used for fusion. The specific process is as follows:

[0126] a. Concatenate the spatiotemporal graph vector sequence and the visual vector sequence respectively, and use a fully connected network to transform the dimensions of the concatenated vectors to obtain a representation vector V with consistent dimensions. graph and V vision ;

[0127] b. Use the Count Sketch mapping function to map V graph and V vision Process and get the corresponding CountSketch vector and

[0128] c. Pair of vectors and Use fast Fourier transform (FFT) to transform to the frequency domain and multiply element by element, and then use inverse fast Fourier transform (IFFT) to obtain the fusion result vector in the time domain

[0129] 2. Multimodal fusion based on cross-modal attention mechanism

[0130] For the spatiotemporal graph vector sequence and the visual vector sequence, this embodiment simultaneously uses a cross-modal attention mechanism for fusion to obtain a fusion vector based on the attention mechanism. The specific operation process is as follows:

[0131] a. Similarly, the spatiotemporal graph vector sequence and the visual vector sequence are concatenated and dimensionally transformed to obtain a representation vector V with the same dimension. graph and V vision ;

[0132] b. Representation vector V of two modalities based on Attention mechanism graph and V vision Fusion is performed to obtain the final fusion vector The specific algorithm is shown in formulas (12)-(14).

[0133]

[0134]

[0135]

[0136] in, are all learnable parameters, and d is the number of attention heads.

[0137] 3. Traffic flow prediction based on multimodal data

[0138] The multimodal fusion result vector of the spatiotemporal graph based on MCB and the road network trajectory graph is Multimodal fusion vector based on cross-modal attention mechanism and the weather factor vector V weather The splicing is performed and the traffic flow is predicted to obtain the traffic flow prediction result Y at the next moment. The specific calculation process is shown in formulas (15)-(17):

[0139]

[0140]

[0141]

[0142] Among them, W 1 ,b 1 ,W 2 ,b 2 are all learnable parameters, γ 1 +γ 2 =1.

[0143] In summary, compared with the existing methods, the present invention proposes a method of using the time-space matrix conversion of vehicle speed to construct a traffic flow spatiotemporal graph with weights, and improves STSGCN to form a weighted-STSGCN model. The method of padding and weight fusion can efficiently extract the structural characteristics of the traffic flow spatiotemporal graph; at the same time, the present invention also provides an image sequence generation method based on the combination of road network and vehicle flow, which can effectively convert road network and trajectory information into visual features. Based on the above characteristics, the present invention proposes a multimodal fusion model based on MCB and Attention, which fuses the graph structural features of the spatiotemporal graph and the visual features of the road network trajectory, and splices the meteorological characteristics with the fused features, so as to achieve high-accuracy traffic flow prediction.

Claims

1. A method for constructing a traffic flow prediction model, characterized in that: The following steps are involved: Step 1: construct a spatiotemporal graph of traffic flow, extract the spatiotemporal graph features of traffic flow, and generate a spatiotemporal graph representation vector; Step 2: construct an image sequence based on the road network, vehicle trajectory and sensor position data, extract features from the image sequence, and obtain a visual representation vector of the image sequence; Step 3: Extract the characteristics of weather factors to form a vector representation of weather factors; Step 4: Use the MCB and Attention dual-channel multimodal fusion model to fuse the spatiotemporal graph representation vector with the visual representation vector of the image sequence to obtain a fusion vector that aligns the graph structure with the visual features, and finally splice the weather factor vector to obtain a traffic flow prediction model. The specific process is as follows: (1) The MCB algorithm is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain the fusion result vector (2) A cross-modal attention mechanism is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain a fusion vector based on the attention mechanism. (3) and the weather factor vector V weather The splicing is performed and the traffic flow is predicted to obtain the traffic flow prediction result Y at the next moment. The specific calculation process is shown in formulas (15)-(17): Among them, W1, b1, W2, b2 are all learnable parameters, γ1+γ2=1, Represents the fusion vector of spatiotemporal graph features and visual features, A fusion vector representing spatiotemporal features, visual features, and weather features.

2. A method for constructing a traffic flow prediction model according to claim 1, characterized in that: When constructing the spatiotemporal graph of traffic flow in step 1, spatial graphs at n different moments are constructed according to the relevant data of the traffic flow sensor. For the spatial graph at n moments, all nodes are connected to themselves at adjacent time steps as the time edges between sensor nodes, thereby obtaining the global spatiotemporal graph of the sensor for n moments. The spatial edge weights of the sensor nodes are The calculation method is shown in formulas (1) and (2): In the above formula, d α,β is the spatial distance between sensor α and sensor β, is the spatial score between sensor α and sensor β; Temporal edge weights of sensor nodes The calculation method is shown in formulas (3) and (4): In the above formula, N is the number of sensors, is the average speed of vehicles in the area, △t is the time interval between two moments, is the temporal edge score of the sensor node.

3. A method for constructing a traffic flow prediction model according to claim 1, characterized in that: In step 1, the improved STSGCN is used to extract features from the spatial graph structure, which specifically includes the spatiotemporal graph serialization based on sliding windows and the STSGCM local feature extraction based on weight fusion, where: (1) Spatiotemporal graph serialization based on sliding windows The global space-time graph is serially segmented by using a sliding window combined with sequence filling to obtain a series of local space-time graphs. (2) STSGCM local feature extraction based on weight fusion Based on STSGCM, the segmented spatiotemporal graph sequences are respectively feature extracted to obtain the corresponding spatiotemporal graph feature sequences; then the features of each spatiotemporal graph sequence are fused based on the weights to finally obtain the output vector sequence, i.e., the spatiotemporal graph representation vector.

4. A method for constructing a traffic flow prediction model according to claim 3, characterized in that: When serializing and segmenting the global space-time graph, a sliding window with a window size of 3 and a step size of 1 is used to serialize and segment the time direction. At the same time, t0 and t n+1 The spatial graph is randomly initialized at every moment to reduce the feature loss of feature extraction; after serialization segmentation, a local spatiotemporal graph with n time steps of 3 is obtained.

5. A method for constructing a traffic flow prediction model according to claim 3, characterized in that: When extracting features from the segmented spatiotemporal graph sequence based on STSGCM, the specific formulas of STSGCM operation are shown in formula (6) and formula (7): H (l) =σ(A′H (l-1) W+b) (6) H i (out) =λ1max(H i (1) ,H i (2) ,...,H i (L) )+λ2H i (L) (7) Where A′ represents the adjacency matrix of the local spatiotemporal graph with a time step of 3, and H (l-1) is the input of the lth graph convolutional layer, W and b are learnable parameters, σ is the activation function, and H i (j) represents the j-th layer graph convolution output vector of the i-th local empty graph sequence, λ1+λ2=1, L is the number of graph convolution layers, H i (out) It is the graph convolution fusion output result of the i-th local empty graph sequence.

6. A method for constructing a traffic flow prediction model according to claim 5, characterized in that: The specific operation of fusing the features of the spatiotemporal graph sequence based on weights is shown in the following formula: in, Represents the j-th layer graph convolution output vector at three moments of the i-th local spatiotemporal graph sequence, ω i-1 +ω i +ω i+1 =1, and substituting it into formula (7) we get: After feature extraction of all local spatiotemporal graph sequences, the output vector sequence is finally obtained 7. A method for constructing a traffic flow prediction model according to any one of claims 1 to 6, characterized in that: In step 3, a weather factor extraction algorithm based on time attenuation is used to extract weather-related factors. The specific process is as follows: (1) In the model training phase, the time series feature vectors of the four weather factors are constructed based on the historical meteorological data of rainfall, temperature, air pressure, and wind speed as the model input data; (2) In the model prediction stage, weather forecast data for a certain time interval in the future are selected as initial data, and a polynomial-based method is used to correct the weather forecast data as the weather factor input data for model prediction. The specific calculation process is shown in formula (11): in, Represent the actual rainfall, temperature, air pressure and wind speed at time c, respectively. They represent the rainfall, temperature, air pressure, and wind speed of the weather forecast at time t, respectively. Respectively represent the rainfall, temperature, air pressure and wind speed of the revised weather forecast at time t, μ c , c=1,2,…,t-1 is the historical weather correction parameter, η is the current weather forecast correction parameter, μ c and η are calculated based on the mean of historical statistical results.

8. A traffic flow prediction method, characterized in that: The following steps are involved: Step 1: Get data Obtain relevant data from traffic flow sensors, road network and vehicle GPS trajectory information, and weather and environmental factor information; Step 2: Based on the data obtained in step 1, a traffic flow prediction model is constructed according to the method described in any one of claims 1 to 7; Step 3: Predict the traffic flow at the next moment based on the constructed prediction model.

9. A traffic flow prediction device, characterized in that: include: A space-time graph representation vector construction module is used to construct a space-time graph of traffic flow based on relevant data of traffic flow sensors, extract the space-time graph features of traffic flow, and generate a space-time graph representation vector; A visual representation vector construction module for image sequences is used to generate road network and trajectory image sequences at different times based on road network data, and to extract features from the generated road network and trajectory image sequences to obtain visual feature representation vector sequences; A vectorized representation building module for weather factors, which is used to extract the features of weather factors and form a vectorized representation of weather factors; And the fusion module is used to fuse the generated spatiotemporal graph representation vector, visual feature representation vector sequence and weather factor vector representation based on the dual-channel fusion model of MCB and Attention to form a traffic flow prediction model and predict the traffic flow at the next moment. The specific process is as follows: (1) The MCB algorithm is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain the fusion result vector (2) A cross-modal attention mechanism is used to fuse the feature vector sequence of the spatiotemporal graph and the visual vector sequence of the road network and trajectory to obtain a fusion vector based on the attention mechanism. (3) and the weather factor vector V weather The splicing is performed and the traffic flow is predicted to obtain the traffic flow prediction result Y at the next moment. The specific calculation process is shown in formulas (15)-(17): Among them, W1, b1, W2, b2 are all learnable parameters, γ1+γ2=1, Represents the fusion vector of spatiotemporal graph features and visual features, A fusion vector representing spatiotemporal features, visual features, and weather features.