A traffic network encoding representation learning method based on masked graph attention mechanism

Abstract

The present invention discloses a traffic network coding representation learning method based on a masked graph attention mechanism, which utilizes the context information extraction capability of the BERT model to expand the adjacent space information extraction order of the graph attention mechanism, so that the model can complete the deep representation of the road network traffic flow data and effectively extract the road network spatial features. The RNERT model adopts a mask pre-training algorithm to learn the common capabilities of road network spatial feature extraction. For the missing data completion task, the RNERT pre-training model is directly used to perform completion calculations; for road network traffic flow prediction, the traffic network gated recurrent network (GRU‑RNERT) represented by the graph attention coding is used to extract the spatiotemporal multi-dimensional features therein.

Description

Technical Field

[0001] The present invention belongs to the field of traffic network traffic flow processing, and in particular relates to a traffic network coding representation learning method based on a mask graph attention mechanism. Background Art

[0002] With the rapid development of domestic urban modernization and the continuous increase in urban population density, traffic congestion has become increasingly prominent, bringing tremendous pressure to social development and becoming an urgent problem to be solved. Improving urban traffic conditions requires traffic planners and managers to take active measures. Scientific and effective guidance and allocation of road network traffic flow will help alleviate urban traffic congestion. In recent years, with the continuous improvement of urban infrastructure, the continuous development of Internet communication technology and the widespread use of positioning equipment, a large number of traffic sensors have been distributed to road facilities and personal terminals, and sensor data has reached a very large cumulative amount. In the era of traffic big data, the processing and analysis technology of traffic big data as an important part of the intelligent transportation system is undoubtedly an indispensable research focus.

[0003] At present, the research on deep learning theory and models in various fields is becoming more and more in-depth and extensive. The continuous development of graph neural networks, traffic flow spatial feature extraction models and sequence representation models has enabled deep learning models to continuously demonstrate remarkable success and great potential in the fields of non-Euclidean space task processing, traffic flow spatiotemporal feature extraction and sequence deep representation. However, there are still some shortcomings or gaps. The road network spatial information extraction model has defects, and the field of traffic flow processing also lacks pre-trained models with common capabilities. In addition, the encoding representation model has been widely promoted and applied in the field of text processing, but it is still blank in the field of traffic flow sequence processing. How to use the sequence representation model and its pre-training method to extract and mine the intrinsic characteristics of road network traffic flow is a problem that needs to be solved by existing technologies.

[0004] Purpose of the Invention

[0005] The purpose of the present invention is to address the difficulties faced by the prior art, and to provide a traffic network coding representation learning method based on a masked graph attention mechanism for the difficulty of spatial feature extraction in road network traffic flow processing. For the road network topology, the spatial adjacency relationship between nodes is regarded as the contextual relationship of the entire graph structure, and the powerful contextual semantic extraction ability of BERT is used to solve the problem of the lack of spatial feature extraction of the road network by the graph attention network. The spatial aggregation ability of the graph attention network is enhanced by using the Transformer coding layer architecture, so that it can aggregate high-order adjacent node information; the Transformer coding layer architecture is endowed with graph data processing capabilities by using the graph attention network, so that it can extract contextual spatial feature information from graph data, and a road network coding representation model with powerful spatial feature extraction capabilities is designed and implemented, and partial mapping calculations are performed on the road network node information. Through the mask pre-training method, the model has a general spatial feature extraction capability, which can be used as a pre-training model in subsequent research to support different traffic tasks and reduce the training cost in different application scenarios, providing effective model support for the difficulties of spatial feature extraction faced in solving the problem of traffic flow modeling in traffic networks. Summary of the invention

[0006] The present invention provides a traffic network coding representation learning method based on a mask graph attention mechanism, which specifically includes the following steps:

[0007] Step 1: Combine the bidirectional encoding representation model and the graph attention network to build a traffic network encoding representation model RNERT based on the masked graph attention mechanism. Use mask pre-training to train the model's general ability to extract road network spatial features to support different traffic tasks.

[0008] Step 2: Use the pre-trained RNERT model to complete the missing values ​​of the traffic network at a certain moment;

[0009] Step 3: Based on RNERT, a traffic network gated recurrent network GRU-RNERT is constructed. RNERT is embedded into the recurrent neural network as a pre-trained spatial feature extraction unit, and participates in the calculation of the gated unit, so that it can extract the spatiotemporal multi-dimensional features of the traffic flow in the traffic network and realize the time series modeling of the traffic flow in the road network.

[0010] Preferably, the step 1 further comprises the following sub-steps:

[0011] Sub-step S11, model input, includes taking the average speed of traffic flow as the model input feature value, the model input is the feature value matrix and adjacency matrix of the road network nodes at a certain moment; normalizing the feature matrix, using the maximum and minimum normalization method to convert the average speed into a decimal in the interval [0,1], and the calculation method is expressed as shown in formula (1):

[0012]

[0013] Sub-step S12, model coding calculation, the road network coding representation model is composed of multiple coding layer Encoders, whose input is aggregated into the current node through multi-layer coding transmission to obtain the node feature matrix with the same dimension as the input and containing the road network spatial information; the node feature matrix refers to the data matrix composed of traffic flow indicators in all sections of the road network at a certain moment, the node refers to the section in the road network, and the node feature value refers to the traffic flow data on the section; the road network adjacency matrix is ​​a 0, 1 matrix, which represents the connection relationship between nodes of different sections, and the corresponding positions of two sections with connection relationship are assigned 1 in the matrix; the coding layer Encoder is composed of a multi-head graph attention layer M-GAT, a residual normalization layer A&LN and a multi-layer perception mechanism feedforward transmission layer MLP; the calculation process of the road network coding representation model includes: the l-th coding layer Encoder l The forward transmission process in is expressed as shown in equations (2)-(5):

[0014] s 1 =M-GAT(h l-1 ,A) (2),

[0015] s 2 =A&LN(h l-1 ,s 1 ) (3),

[0016] s 3 =MLP(s 2 ) (4),

[0017] h l =A&LN(s 2 ,s 3 ) (5),

[0018] Among them, s 1 、s 2 ∈R n×K 、s 3 ∈R n×1 is the intermediate transfer variable of the encoding layer; h l-1 ∈R n×1 is the output of the previous encoding layer; h l ∈R n×1 is the output of the current coding layer; A∈R n×n is the road network adjacency matrix; K is the number of heads of the multi-head attention mechanism; M-GAT(·) is the multi-head attention spatial information aggregation; A&LN(·) is the residual normalization calculation; MLP(·) is the feedforward transfer calculation;

[0019] Sub-step S13, model mask pre-training, perform random masking operations on the input, set the eigenvalues ​​to 0 or null at a ratio of 15%, forward propagate the masked input data, and obtain an output with the same dimension; use the loss function to calculate the error loss between the true value of the masked area and the model output value, solve the gradient, and back-propagate the error until the model converges.

[0020] Preferably, the calculation process of the multi-head graph attention layer M-GAT is expressed as shown in formulas (6)-(8):

[0021]

[0022] Among them, e k ij is the node importance between nodes i and j in the k-th layer graph attention network; is the eigenvalue of node i and node j; is the shared weight matrix of the k-th layer graph attention network; a k ij is the importance coefficient between nodes i and j in the k-th layer graph attention network; N i is the adjacent set of node i; a∈R 2 is the concatenated vector; K is the number of heads of the multi-head attention mechanism; h′ i ∈R K is the node feature value after attention calculation and concatenation; Leaky ReLU(·), Softmax(·), and f(·) are activation functions. In the model design, σ(·) takes the Leaky ReLU(·) activation function;

[0023] The forward transfer process of the residual normalization layer of the first layer in the coding layer is shown in equations (9)-(12):

[0024] a=(h l-1 W A&LN +s 1 )(9),

[0025]

[0026] Where a∈R n×K is the residual value of the output model of the encoding layer input and the multi-head attention layer M-GAT; W A&LN ∈R K is the weight vector of the residual module; K is the number of multi-head attention heads; n can be regarded as the number of network nodes in the previous layer. The data distribution changes in this dimension, and the layer normalization is processed according to the dimension n; μ, σ∈R Kis the mean and variance of a; ε takes zero value; g and b are gain and bias values, and the effect of layer normalization is guaranteed by learning this parameter; s 2 ∈R n×K is the output of the residual normalization layer; the calculation formula of the residual normalization layer of the second layer of the coding layer is the same as that of the first layer;

[0027] The calculation method of the feedforward transfer layer is shown in formula (13):

[0028]

[0029] Among them, W MLP1 ∈R F×K , W MLP2 ∈R F is the weight matrix of the fully connected layer; b MLP1 、b MLP2 is the bias vector; F is the number of hidden nodes in the feedforward transfer layer, which is a model hyperparameter and is optimized through experiments; σ(·) is the activation function, and LeakyReLU(·) is used in the model.

[0030] Preferably, the step 3 further comprises the following sub-steps:

[0031] Sub-step S31, model input, the GRU-RNERT model is a time series with a step length of L, the extracted node feature values ​​are also normalized, and the maximum and minimum normalization method is used to convert the average speed into a decimal in the interval [0,1];

[0032] Sub-step S32, model loop calculation; The model framework of the GRU-RNERT model combines the gated recurrent network and the RNERT model, extracts spatial information extraction capabilities through the RNERT unit, and embeds the model into the model loop calculation process, so that the memory unit simultaneously resets and memorizes the spatiotemporal features, and memorizes the node feature values, adjacency matrix and model output of the previous moment in the model input sequence. After the internal memory unit of the model memorizes the spatiotemporal information of the input sequence, it continues to output it as a hidden vector h until the entire sequence is input. The transfer formula is expressed as shown in formulas (14)-(15):

[0033] h 1 =GRU-RNERT(x 1 ,h 0 ,A)

[0034] h 2 =GRU-RNERT(x 2 ,h 1 ,A)

[0035] h 3 =GRU-RNERT(x 3,h 2 ,A) ...

[0037] h L =GRU-RNERT(x L ,h L-1 ,A) (14),

[0038] y L =Leaky ReLU(W o h L ) (15),

[0039] Among them, x 1 、x 2 、x 3 With x L ∈R n×1 is the node feature matrix in the sequence, L is the length of the input sequence; h 0 、h 1 、h 2 、h L-1 With h L ∈R n×1 is a hidden vector that memorizes the spatiotemporal features of the traffic flow sequence, h 0 Take zero value; W o Output weight matrix for the model; A∈R n×n is the road network adjacency matrix; L ∈R n×1 is the final output of the model; GRU-RNERT(·) is the forward calculation process of the GRU-RNERT model;

[0040] Sub-step S33, model time series modeling task training, GRU-RNERT is implemented based on the RNERT pre-training model, so the model training is divided into two parts: RNERT unit pre-training and GRU-RNERT specific task training; GRU-RNERT specific task training adopts time series data prediction tasks, and trains the model's road network traffic flow sequence spatiotemporal feature extraction capability by extrapolating and predicting road network time series data, wherein the RNERT unit parameters are not fixed, and its pre-training results are regarded as the initial parameter values, and all model parameters are trained and adjusted; the second step of model training adopts a supervised gradient descent algorithm, and inputs a road network eigenvalue matrix time series with a step size of L, extrapolates the feature matrix of the next moment of the sequence, and adds the model output to the end of the sequence as input for the next extrapolation calculation, and finally calculates the error function value between the model output and the real data and performs back propagation, updates the model parameters, and trains the model to extract spatial and temporal features from road network time series data.

[0041] Preferably, in sub-step S32, a soft attention mechanism is introduced to calculate the final output of the model, and the final output of the GRU-RNERT model is expressed as shown in equations (16)-(17):

[0042] h′ L =Att(h 1 ,h 2 ,h 3 ,...,h L-1 .h L ) (16),

[0043] y L =Leaky ReLU(W o h′ L ) (17), Among them, the specific calculation process of the GRU-RNERT model is shown in equations (18)-(20):

[0044]

[0045] Among them, e kL is the importance between the hidden vectors at two moments in the sequence, calculated by performing a dot product between the two; a kL is the attention coefficient between hidden vectors. The hidden vectors are weighted and summed according to the attention coefficient, and the final weighted calculation is made on the information contained in the historical input to obtain the final output of the model.

[0046] Preferably, when pre-training the traffic network encoding representation model RNERT to fit the spatial law of the road network, the mean square error is selected as the loss function, and the MSE calculation formula is expressed as shown in formula (21):

[0047]

[0048] Where n is the total number of samples; With y i are the model output and sample true value of the corresponding positions respectively.

[0049] Preferably, the specific transmission of the GRU-RNERT model is shown in equations (22)-(27):

[0050] r′=W r (x t ||h t-1 ) (twenty two),

[0051] r=Sigmoid(RNERT(r′,A)) (23),

[0052] z′=W z (x t ||ht-1 ) (twenty four),

[0053] z=Sigmoid(RNERT(z′,A)) (25)

[0054] h′ t =tanh(RNERT(W h (x t ||(r⊙h t-1 )),A)) (26),

[0055] h t =z⊙h t-1 +(1-z)⊙h′ t (27),

[0056] Among them, W r , W z , W h is the weight matrix, which performs linear calculations with the concatenated matrix of the input and hidden vectors to change the dimension of the concatenated matrix so that it can meet the input requirements of the RNERT unit; t ∈R n×1 is the node feature matrix at time t in the input sequence, A∈R n×n is the road network adjacency matrix; r∈R n×1 To reset the gate, reset the hidden value of the previous moment; z∈R n×1 is the update gate, which determines the proportion of the previous hidden value and the current temporary hidden value in the output; r′ and z′∈R n×1 It is the temporary value of resetting the gate and updating the gate; h t-1 ∈R n×1 is the hidden value of the previous moment; h t is the hidden value at the current moment; h′ t is the temporary hidden value at the current moment; ⊙ is the Hadamard product, that is, the multiplication of corresponding elements of the matrix; Sigmoid(·) and tanh(·) are activation functions; RNERT(·) is the forward calculation of the RNERT unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a structural schematic diagram of the RNERT model of the present invention.

[0058] Figure 2 Schematic diagram of the principle of the mask training method of the present invention.

[0059] Figure 3 Schematic diagram of the calculation principle of the GRU-RNERT model of the present invention.

[0060] Figure 4 Schematic diagram of the GRU-RNERT cyclic calculation process of the present invention.

[0061] Figure 5 Schematic diagram of the GRU-RNERT cycle calculation process for the present invention using the soft attention mechanism

[0062] Figure 6a -c is the mean square error dot-line graph under different hyperparameters of the present invention.

[0063] Figure 7 This is a comparison chart of the true value of the mask node and the output results of the mask node model at different times for the "Wanliu Bridge-Lize Bridge" in Beijing.

[0064] Figure 8a -b is the experimental error dot-line graph under different data missing rates.

[0065] Fig. 9 It is a dot-line diagram of the experimental error under different data missing rates.

[0066] Fig.10a -c is a comparison chart of model completion effects under different data missing rates.

[0067] Fig.11 This is a dotted line graph of the GRU-RNERT model hyperparameter tuning experimental results.

[0068] Fig.12a -c is a dot-line graph of the average error of the results with different prediction step sizes.

[0069] Fig.13 It is a box plot of the mean absolute error distribution of different prediction step sizes of the GRU-RNERT model of the present invention.

[0070] Fig.14 It is a box plot of the average absolute error distribution of different prediction step sizes of each model of the present invention.

[0071] Fig.15a -d is a comparison chart of the results of different prediction step sizes of each model of the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Those skilled in the art will appreciate that the step numbers used herein are only for convenience of description and are not intended to be a limitation on the order in which the steps are performed. The terms used in the present specification are only for the purpose of describing a specific embodiment and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms of "one", "an" and "the" may include plural forms unless the context clearly indicates otherwise. The term "and / or" refers to any combination of one or more of the items listed in association and all possible combinations, and includes these combinations. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0074] Figure 1 It is a structural diagram of the RNERT model of the present invention. As shown in the figure, the input of the RNERT model is the node eigenvalue matrix and the road network adjacency matrix, and the output is the updated feature matrix. The node feature matrix refers to a data matrix composed of traffic flow indicators in all sections of the road network at a certain moment. It only represents the traffic flow eigenvalue at that moment, and cannot describe the spatial relationship. The node is the section in the road network, and the node eigenvalue is the traffic flow data on the section. The road network adjacency matrix is ​​a 0, 1 matrix, which represents the connection relationship between nodes in different sections. Two sections with a connection relationship will be assigned a value of 1 at the corresponding position in the matrix. The feature matrix is ​​normalized, and the maximum and minimum normalization method is used to convert the average speed into a decimal in the interval [0,1]. The calculation method is expressed as shown in formula (1):

[0075]

[0076] The model consists of multiple encoding layers (Encoder). The input is transmitted through multiple layers of encoding to aggregate the spatial features of multi-order adjacent nodes into the current node, and the input traffic flow feature matrix is ​​characterized and calculated, and finally a node feature matrix with the same dimension as the input and containing road network spatial information is obtained. In the process from input to output, the model completes the deep extraction of data spatial distribution features from the adjacency matrix and aggregates them into the road network node feature values, realizing the representation from simple multi-dimensional vector input to the feature matrix that aggregates spatial features, to support the solution of subsequent traffic tasks.

[0077] The model encoding layer consists of a multi-head graph attention network (M-GAT), a residual normalization layer (Add&Layer Norm, A&LN) and a multi-layer perceptron feedforward transfer layer (MultilayerPerceptron, MLP). The input first performs a higher-order aggregation of the road network spatial information through the multi-head attention mechanism, and then performs layer normalization calculation through the residual normalization layer to ensure that the sample distribution remains unchanged. Finally, linear calculation and nonlinear activation are performed through the feedforward transfer layer to increase the model depth, improve the model fitting ability, and adapt to the nonlinear law of the data.

[0078] The model performs multiple encoding calculations on the input, and each encoding causes the node to aggregate higher-order adjacent node information. After multiple encodings, the spatial features are fully extracted, solving the defect that the graph attention network cannot extract high-order spatial features. The dimension of the model input remains unchanged during the transmission of each encoding layer, and the final output result can be regarded as a node feature matrix that has aggregated the spatial information of the road network.

[0079] The specific calculation process of the model encoding layer is further explained. The first layer of the encoding layer Encoder l The forward transmission process in is expressed as shown in equations (2)-(5):

[0080] s 1 =M-GAT(h l-1 ,A) (2),

[0081] s 2 =A&LN(h l-1 ,s 1 ) (3),

[0082] s 3 =MLP(s 2 ) (4),

[0083] h l =A&LN(s 2 ,s 3 ) (5),

[0084] Among them, s 1 、s 2 ∈R n×K 、s 3 ∈R n×1 is the intermediate transfer variable of the encoding layer; h l-1 ∈R n×1 is the output of the previous encoding layer; h l ∈R n×1 is the output of the current coding layer; A∈R n×nis the road network adjacency matrix; K is the number of heads of the multi-head attention mechanism; M-GAT(·) is the multi-head attention spatial information aggregation; A&LN(·) is the residual normalization calculation; MLP(·) is the feedforward transfer calculation.

[0085] Further explanation of the multi-head graph attention layer, residual normalization layer, and feedforward transfer layer in the encoding layer:

[0086] 1) Multi-head graph attention layer

[0087] As the core unit in the model and encoding layer, the multi-head graph attention layer is responsible for the representation calculation of the input matrix, reading the road network topological relationship from the road network adjacency matrix, performing weighted calculation on the node feature values ​​according to the upstream and downstream adjacency relationship of the road section, and updating and iterating the input data, thereby completing the process of extracting spatial features from the adjacency matrix to the node feature matrix. As the input is transmitted in multiple encoding layers, the node feature values ​​are continuously represented and calculated to achieve the aggregation of multi-order spatial information.

[0088] The core of the multi-head attention layer is the multi-head graph attention mechanism, which is used to extract the spatial features of the road network and iterate the node feature values ​​to aggregate the spatial information. Through multiple independent attention layers, the outputs are calculated separately and spliced ​​or averaged, and the different inherent characteristics are processed separately to capture the spatial features more comprehensively. The calculation process is shown in equations (6)-(8):

[0089]

[0090] Among them, e k ij is the node importance between nodes i and j in the k-th layer graph attention network; is the eigenvalue of node i and node j; k

[0091] is the shared weight matrix of the k-th layer graph attention network; a k ij is the importance coefficient between nodes i and j in the k-th layer graph attention network; N i is the adjacent set of node i; a∈R 2 is the concatenated vector; K is the number of heads of the multi-head attention mechanism; h′ i ∈R K is the node feature value after attention calculation and concatenation; Leaky ReLU(·), Softmax(·), f(·) are activation functions, and σ(·) in the model design takes the Leaky ReLU(·) activation function.

[0092] The multi-head attention layer transforms the model input from a vector to a high-dimensional matrix through vector concatenation, enhancing the information expression ability of the intermediate values ​​and enabling the model to better fit the complex laws inherent in the data set.

[0093] 2) Residual Normalization Layer

[0094] The residual normalization layer solves the degradation and internal covariate shift problems that may exist in the model through residual and normalization calculations, improves the model accuracy, and reduces the difficulty of model training. The forward transmission process of the first residual normalization layer in the encoding layer is shown in equations (9)-(12):

[0095] a=(h l-1 W A&LN +s 1 )(9),

[0096]

[0097]

[0098] Where a∈R n×K W is the residual value of the encoding layer input and the multi-head attention layer output model; A&LN ∈R K is the weight vector of the residual module; K is the number of multi-head attention heads; n can be regarded as the number of network nodes in the previous layer. The data distribution changes in this dimension, and the layer normalization is processed according to the dimension n; μ, σ∈R K is the mean and variance of a; ε is a small number to prevent division by zero, which is ignored here and takes zero value; g and b are gain and bias values, and the effect of layer normalization can be guaranteed by learning this parameter; s 2 ∈R n×K is the output of the residual normalization layer. The calculation formula of the second layer of the coding layer is the same as that of the first layer.

[0099] 3) Feedforward transfer layer

[0100] The feedforward transfer layer transforms the input dimension in the form of a fully connected layer, increasing the model complexity and depth, improving the model fitting ability, and ensuring that the input dimension of the next encoding layer meets the requirements. The calculation method is shown in formula (13):

[0101]

[0102] Among them, W MLP1 ∈R F×K , W MLP2 ∈R F is the weight matrix of the fully connected layer; b MLP1 、b MLP2is the bias vector; F is the number of hidden nodes in the feedforward transfer layer, which is a model hyperparameter and is optimized through experiments; σ(·) is the activation function, and LeakyReLU(·) is used in the model.

[0103] After having the structural foundation, the road network data set is used for pre-training, so that the model can fully exert its spatial extraction ability and highly fit the spatial laws of the road network. In order to better enable the model to fit the spatial laws of the traffic road network, the model is trained using regression tasks. In order to achieve better model convergence effect, the mean square error is selected as the loss function. The MSE calculation formula is shown in the formula:

[0104]

[0105] Where n is the total number of samples; With y i are the model output and sample true value of the corresponding positions respectively.

[0106] As a pre-training model, in order to make the RNERT model highly fit the spatial characteristics of the road network, fully extract the spatial characteristics of road network traffic flow when solving specific traffic problems, and fully verify the spatial feature extraction ability of the model, this paper adopts a self-supervised mask pre-training method, which is not oriented to specific downstream tasks, avoids the influence of specific tasks on model training, and trains and verifies the general ability of the model in spatial feature extraction.

[0107] Figure 2 Schematic diagram of the principle of the mask training method of the present invention. Figure 2 As shown, the mask training method refers to performing a random mask operation on the input, setting a certain number of eigenvalues ​​to 0 or null in proportion, forward propagating the input data that has undergone the mask operation, and obtaining an output with the same dimension. The specific mask operation of the present invention on the node is to set the node eigenvalue to zero, that is, to set the average speed of the corresponding road section to zero. According to the specific forward transmission process of the model, it can be known that the process of aggregating the road network spatial information in the coding layer of the RNERT model occurs in the multi-head graph attention calculation. After the feature matrix involved in the calculation is randomly masked, when the selected node calculates the node importance and attention coefficient with the remaining road network nodes with an adjacent relationship and performs the final weighted addition, since the selected node has been set to zero at this moment, for the remaining nodes, the zero-value node does not participate in the final weighted calculation, and for the selected node, the eigenvalues ​​of the remaining nodes will still be aggregated to this position. Therefore, the method of setting the node feature value to zero in the mask training of the RNERT model will not cause additional impact on the spatial information aggregation of the remaining nodes, and at the same time ensure that the spatial information aggregation process of the masked nodes proceeds smoothly.

[0108] The mask training method of the RNERT model can be used as a pre-training model to support specific downstream traffic tasks. Mask training is not aimed at specific traffic problems, but only trains the spatial feature capabilities of the model. When solving specific traffic problems, further fine-tuning of the model parameters to adapt to the task can greatly reduce the training cost and sample size requirements in the model application, and improve its performance in various tasks.

[0109] In mask training, the selection of the feature matrix mask rate also needs to be considered in many aspects. The purpose of training the model using mask pre-training is to enable the model to extract spatial features, rather than to solve specific traffic tasks. If the mask rate is set too small, the nodes cannot be evenly distributed in the entire road network space under the premise of random mask node selection. Training on this basis will cause the model to focus too much on the restoration and completion of a small number of nodes, and it will not be able to fully extract the spatial topological structure of the entire road network. In this case, when the model extracts the spatial structure from the road network adjacency matrix and aggregates spatial information in the node feature matrix, the spatial information will be overly tilted towards a small number of nodes. The model after training cannot be used as a pre-training module to provide sufficient and strong support for the extraction of spatial features in specific traffic tasks. When the mask rate is set too large, the information loss caused to the data set is too large, the model cannot extract enough spatial laws from the incomplete feature matrix, and the model parameters cannot be further trained and iterated, resulting in the trained model not fitting the intrinsic spatial features of the data set well enough, and it also cannot provide strong support for the subsequent solution of specific traffic tasks. Therefore, the present invention considers various factors comprehensively and takes specific experimental results into consideration, and sets the mask rate of input data to 0.15 in mask pre-training.

[0110] (2) Research on data completion based on mask image attention encoding representation

[0111] As the foundation of intelligent transportation systems, the collection of traffic data will be affected by various uncertain factors in practical applications. These factors may interfere with data collection, resulting in missing or erroneous traffic data, which brings various challenges to data collection and analysis. At present, the problem of missing traffic data is still very serious. This phenomenon of missing or erroneous data also brings challenges to subsequent research and applications based on traffic data. The lack of traffic data often affects the accuracy and effectiveness of traffic management and application. At the same time, missing data may also cause many data analysis methods to lose their usability.

[0112] For filling missing values ​​in traffic data, the current mainstream methods are mostly based on data recovery based on multi-time traffic flow sequences, which requires capturing features from traffic flow time series data to calculate or generate missing values. When there is a situation where the missing data is relatively scattered, the traffic flow data of a certain section is seriously missing, and it is impossible to provide sufficient features for data completion from the perspective of time series, it is necessary to consider estimating the missing values ​​based on the spatial structure of the road network using the data of the remaining sections at the current moment and the adjacency relationship between sections. When faced with the situation where only single-time road network traffic flow data is missing, most current methods cannot calculate and complete the missing values ​​using only the values ​​of the remaining nodes in the road network, which greatly limits the application scenarios of data completion methods.

[0113] According to the RNERT model design and implementation and model experiments, the RNERT model, as a pre-training model, has fully learned the extraction of road network spatial features through random mask training. It can represent the connection relationship of road sections in the road network in the road network node eigenvalue matrix, so that the eigenvalue matrix is ​​changed from an unrelated multi-dimensional vector to an eigenvector that aggregates the spatial feature information of the road network. This effective spatial feature extraction capability undoubtedly provides a feasible solution for the completion of missing values ​​in road network data.

[0114] According to the detailed description of the model training method, it can be learned that in the mask training adopted by the RNERT model, the model input is the masked road network traffic flow feature matrix, and the output is the traffic flow feature matrix characterized by the mask node. When the model is subjected to loss function calculation and error back propagation, the model error is calculated by the original value of the mask node and the representation value of the mask position, so as RNERT's learning of spatial feature extraction capabilities continues to deepen during the mask pre-training process, the model can also achieve data restoration of the mask node. This is very similar to the task of completing missing values ​​in the road network, because RNERT already has a good mask value restoration effect after parameter tuning during the training process, and in order to show the performance of RNERT as a pre-training model in solving specific traffic tasks, the present invention directly uses the trained RNERT model to complete the missing values ​​of the road network node traffic flow at a single moment.

[0115] (3) Gated Recurrent Network for Traffic Networks Based on Graph Attention Encoding

[0116] Recurrent neural networks have excellent performance in fitting time series features, but when solving the problem of deep learning modeling of traffic network traffic flow, simply using the road network data at each moment in vector form as the model input will cause the loss of its road network topology information, and the road network data will only be treated as multi-node data. In order to take into account the mining of data space rules, it is necessary to use a model architecture with powerful spatial feature extraction capabilities. The RNERT model's powerful spatial information extraction capabilities and its unique mechanism for deep spatial information aggregation of input node feature matrices enable it to be used to solve the problem that traditional recurrent neural network memory units cannot memorize spatial information. The present invention uses the RNERT pre-trained model as an embedded unit to participate in the calculation process of its gating unit. While extracting spatial information from the adjacency matrix and aggregating it into the input matrix, it incorporates spatiotemporal multidimensional factors into the calculation of the gating unit to construct a model that can extract spatiotemporal multidimensional features of road network traffic flow data.

[0117] Figure 3 This is a schematic diagram of the calculation principle of the GRU-RNERT model. Figure 3 As shown in the figure, the model framework of the GRU-RNERT model combines the design features of the gated recurrent network and the RNERT model. Through the powerful spatial information extraction capability of the RNERT unit, the model is embedded into the model's cyclic calculation process, so that the memory unit no longer only memorizes the time features in the sequence like the traditional recurrent neural network, but resets and memorizes the spatiotemporal features at the same time.

[0118] The input of the model is the road network node feature matrix, spatial adjacency matrix and hidden value output by the model at the previous moment. The hidden matrix and input matrix of the previous moment are horizontally spliced, and the temporary value after dimension restoration is obtained by linear multiplication with the weight matrix. The model reset gate and update gate are obtained by RNERT unit calculation and activation function. Among them, the reset gate is responsible for determining how much of the hidden value of the previous moment can participate in the calculation of the current moment, and further calculates the temporary hidden value of the current moment with the current input; the update gate is responsible for the output calculation of the final model hidden value, and determines the respective proportion of the hidden value and temporary value of the previous moment in the output. The larger the update gate, the greater the proportion of the hidden value of the previous moment in the final output. After obtaining the hidden value of the current moment, continue to loop the calculation and use it as the input of the model of the next moment until all moments in the input sequence are calculated.

[0119] Because of the participation of the RNERT unit, the input of the GRU-RNERT model is no longer a simple high-dimensional matrix data, but a road network traffic flow data with a topological spatial structure. The RNERT unit participates in the calculation process of the model reset gate, update gate and temporary hidden value, and updates the eigenvalues ​​to aggregate the contextual spatial information within a certain adjacent range, so that the model fully considers the spatial structure of the road network and the spatial characteristics in the data when memorizing and updating iterative sequence information. Therefore, the model is no longer a simple time series data processing model, but a neural network that focuses on spatiotemporal multi-feature extraction and regularity fitting. The specific transmission of the GRU-RNERT model is shown in equations (22)-(27):

[0120] r′=W r (x t ||h t-1 )(twenty two),

[0121] r=Sigmoid(RNERT(r′,A))(23),

[0122] z′=W z (x t ||h t-1 )(twenty four),

[0123] z=Sigmoid(RNERT(z′,A))(25)

[0124] h t ′=tanh(RNERT(W h (x t ||(r⊙h t-1 )),A))(26),

[0125] h t =z⊙h t-1 +(1-z)⊙h t ′(27),

[0126] Among them, W r , W z , W h is the weight matrix, which performs linear calculations with the concatenated matrix of the input and hidden vectors to change the dimension of the concatenated matrix so that it can meet the input requirements of the RNERT unit; t ∈R n×1 is the node feature matrix at time t in the input sequence, A∈R n×n is the road network adjacency matrix; r∈R n×1 To reset the gate, reset the hidden value of the previous moment; z∈R n×1 is the update gate, which determines the proportion of the previous hidden value and the current temporary hidden value in the output; r′ and z′∈R n×1It is the temporary value of resetting the gate and updating the gate; h t-1 ∈R n×1 is the hidden value of the previous moment; h t is the hidden value at the current moment; h′ t is the temporary hidden value at the current moment; ⊙ is the Hadamard product, that is, the multiplication of corresponding elements of the matrix; Sigmoid(·) and tanh(·) are activation functions; RNERT(·) is the forward calculation of the RNERT unit.

[0127] Figure 4 This is a schematic diagram of the GRU-RNERT cycle calculation process. Figure 4 As shown, for the node feature values, adjacency matrix and model output of the previous moment in the model input sequence, the internal memory unit of the model memorizes the temporal and spatial information of the input sequence and continues to output it as the hidden vector h until the entire sequence is input. As shown in formulas (14)-(15):

[0128] h 1 =GRU-RNERT(x 1 ,h 0 ,A)

[0129] h 2 =GRU-RNERT(x 2 ,h 1 ,A)

[0130] h 3 =GRU-RNERT(x 3 ,h 2 ,A) ...

[0132] h L =GRU-RNERT(x L ,h L-1 ,A) (14),

[0133] y L =Leaky ReLU(W o h L ) (15),

[0134] Among them, x 1 、x 2 、x 3 With x L ∈R n×1 is the node feature matrix in the sequence, L is the length of the input sequence; h 0 、h 1 、h 2 、h L-1 With h L ∈R n×1is a hidden vector that memorizes the spatiotemporal features of the traffic flow sequence, h 0 Take zero value; W o Output weight matrix for the model; A∈R n×n is the road network adjacency matrix; L ∈R n×1 is the final output of the model; GRU-RNERT(·) is the forward calculation process of the GRU-RNERT model.

[0135] In order to solve the problem of information loss caused by too long sequences in traffic flow prediction, this section further improves the cycle process of the GRU-RNERT model. Figure 5 This is a schematic diagram of the GRU-RNERT cycle calculation process that introduces the soft attention mechanism. Figure 5 As shown in Figure 1, the soft attention mechanism is introduced to calculate the final output of the model, which improves the final performance of the GRU-RNERT model and provides a more reliable model method for traffic flow prediction. The final output of the GRU-RNERT (Att) model is shown in equations (16)-(17):

[0136] h′ L =Att(h 1 ,h 2 ,h 3 ,...,h L-1 .h L ) (16),

[0137] y L =Leaky ReLU(W o h′ L ) (17).

[0138] The specific calculation process of Att(·) is shown in equations (18)-(20):

[0139]

[0140]

[0141] Among them, e kL is the importance between the hidden vectors at two moments in the sequence, calculated by performing a dot product between the two; a kL is the attention coefficient between hidden vectors. The hidden vectors are weighted and summed according to the attention coefficient, and the final weighted calculation is made on the information contained in the historical input to obtain the final output of the model.

[0142] GRU-RNERT is implemented based on the RNERT pre-training model, so the model training is divided into two steps, namely RNERT unit pre-training and GRU-RNERT specific task training, in order to improve training efficiency, reduce training time and verify the common capabilities of the RNERT model. The specific training task of the GRU-RNERT model adopts the time series data prediction task, which is also a training method commonly used by traditional recurrent neural networks. Through the extrapolation and prediction of road network time series data, the model's ability to extract spatiotemporal features of road network traffic flow sequences is trained. Through the model training results and test set performance, the superiority of GRU-RNERT in road network traffic flow prediction and the gain effect of the RNERT pre-training model in road network traffic flow modeling are verified.

[0143] In the specific task training of GRU-RNERT, the parameters of the RNERT unit are not fixed, but the pre-training results are regarded as the initial values ​​of the parameters, and all the parameters of the model are trained and adjusted. On the one hand, this training method reduces the cost of model training. On the other hand, continuing to adjust the parameters of the pre-trained RNERT unit can make the model fit the prediction task better and avoid the excessive influence of mask training on the unit in pre-training, which makes the model insensitive to the prediction task. In the second step, the model training adopts a supervised gradient descent algorithm, inputs a time series of road network eigenvalue matrix with a step size of L, extrapolates the feature matrix of the next moment of the sequence, and adds the model output to the end of the sequence as input for the next step of extrapolation calculation. Finally, the error function value between the model output and the real data is calculated and back-propagated, the model parameters are updated, and the training model is capable of extracting spatial and temporal features from road network time series data. In the training process, the Adam optimizer is introduced to optimize the model convergence process. Unlike unsupervised learning, supervised learning requires giving the model input feature value a label value, that is, the real value corresponding to the output, and constructing a training data set with a pair of input features and label values ​​as sample data pairs.

[0144] Example

[0145] All data used in this embodiment are traffic flow data of Beijing’s road network. Each piece of traffic flow data contains five attribute fields, namely time, section name, section length, congestion index and average speed. The collection time granularity is 5 minutes. The collection time period is from December 7, 2019 to January 17, 2020, and the number of sections is 250.

[0146] In order to use the graph structure to represent the traffic flow of the existing traffic road network, each road section in the road network is regarded as a node. The extraction of the road network feature matrix will be carried out from two aspects. On the one hand, the node eigenvalue matrix is ​​constructed, which contains the average traffic flow speed of all road sections in the road network at a certain moment; on the other hand, the adjacency matrix of the road network topological relationship is constructed, which contains the interconnection relationship between all road sections in the road network. The two matrices jointly represent the traffic flow data of the traffic road network under the graph structure.

[0147] (1) RNERT model pre-training experiment

[0148] Through a large number of hyperparameter value experiments, the error values ​​of the validation set under different hyperparameter combinations are compared, and the experimental MSE results within a certain value range where the optimal result is located are displayed in a graph. The specific values ​​of the experimental results are shown in Table 3. Figure 6a , Figure 6b , Figure 6c It is a dotted line graph of the mean square error under different hyperparameters, such as Figure 6a , Figure 6b , Figure 6c As shown, a dotted line graph is drawn for the MSE in the table to facilitate comparative analysis of the experimental results.

[0149] As an indispensable hyperparameter in artificial neural network training, the learning rate determines the step size of the iterative update of model parameters. The correct setting of the learning rate will enable the model to find the global optimal solution faster.

[0150] Table 3 Mean square error results of RNERT model hyperparameter tuning experiment

[0151]

[0152]

[0153] like Figure 6b As shown in , the appropriate learning rate LR of the model is 0.005. In addition to the learning rate, the experiment also performs hyperparameter tuning on the number of graph attention heads K and the number of hidden nodes F in the feedforward transfer layer. Combining the experimental results and specific analysis, in order to make the model find the complexity balance point, achieve convergence at the global optimal solution, and achieve the best spatial feature extraction effect, the model's hyperparameters LR, K, and F are set to 0.005, 5, and 6 respectively.

[0154] After completing the model hyperparameter tuning, the optimal hyperparameter combination is used for model training to complete the learning of road network spatial feature extraction, and the test set data is used to make a preliminary judgment on the model effect. The model test set adopts the same mask rate as the training set, and finally obtains the average MAE of all outputs of 3.0695km / h, MSE of 18.9827, and MAPE of 6.1789%. It shows that after training, the model can deeply extract the spatial features of the road network and aggregate the information on the mask node, realize the generation of mask values, and achieve the expected effect of mask pre-training.

[0155] After masking the single-time road network data in the test set, the corresponding output value is obtained as the model input. In order to evaluate the model training results more intuitively, Figure 7 This is a comparison chart of the real value of the mask node and the output result of the mask node model at different times for the "Wanliu Bridge-Lize Bridge" in Beijing. Figure 7 As shown in the figure, the "Wanliu Bridge-Lize Bridge" section with the smallest section number is selected from all masked sections, and the actual value of the average traffic flow speed of this section at different times in the test set is compared with the model output value. It can be seen that the output value of the model at the masked node is very close to the actual value of the node traffic flow, which means that the masked zero-value road network node can fully extract and aggregate the traffic flow information in the high-order adjacent sections to restore the traffic flow data of this section. In addition, even if the model input is a single-time road network feature matrix, there are only road network spatial features in the input without traffic flow sequence time features, but after arranging the model output in chronological order, it can still fit the complete road section traffic flow feature value time change trend. This fully demonstrates that the spatial dependency in the road network is very powerful. The traffic flow information of the masked road section can be accurately inferred solely by the spatial features of the road network at the same time. It also shows that the RNERT model has acquired powerful spatial feature extraction and aggregation capabilities for traffic road networks after training and learning. It can fully and deeply mine the spatial dependencies in the road network connection topology, aggregate the road section connection relationships in the road network adjacency matrix into the traffic flow node feature matrix, and complete the spatial feature representation of the traffic flow feature matrix of the road network at a single time.

[0156] In summary, the model performed well in the test set, indicating that the RNERT model has a strong ability to extract road network spatial features and can deeply mine and fit the complex spatial dependencies and rules in road network traffic flow.

[0157] (2) Experiment on completing missing data of traffic flow in traffic network

[0158] In the experiment, the masking rates of 5%, 10%, 15%, 20%, 30%, 40%, 50%, 60% and 70% were used to randomly mask the road network node data in the test set, simulating the problem of different degrees of traffic flow data missing in real situations. At this time, the masking rate can also be called the data missing rate. The masked data is input into the RNERT pre-trained model, and the missing values ​​are supplemented by the road network traffic flow data at the same time. The average error of the model experimental results under different data missing rates is shown in Table 5.

[0159] Table 5 Average error of completion results under different data missing rates

[0160]

[0161]

[0162] Figure 8a , Figure 8b It is a dot-line graph of experimental error under different data missing rates. As shown in the figure, Figure 8a , Figure 8b The MSE, MAE, MAPE, and RMSE are plotted as dot-line graphs. It is not difficult to see that when the data missing rate gradually increases from 5% to 70%, the error of the test set decreases first and then increases, but it is generally positively correlated with the missing rate.

[0163] When the data missing rate is between 5% and 20%, the error values ​​are at a low level, the difference between them is small, and the trend of the dot-line graph is relatively stable, but at the same time, it has a trend of decreasing first and then increasing, and the error value reaches a minimum at 15%. This is because when the data missing rate is small, the number of data missing nodes is small, and the distribution in the entire road network will not have much impact on the overall spatial information of the road network. By using the high-order adjacent nodes around the missing nodes, the missing values ​​can be restored or completed more accurately. From 5% to 20%, the data missing caused by each missing rate is relatively limited, so the model can achieve a good completion effect, so that the final error value is at a low level, and there will not be much difference under each missing rate. In addition, affected by the value of the data mask rate during the pre-training of the RNERT model, the model will perform slightly better than the other missing value cases when the data missing rate is 15%, so the dot-line graph will show a trend of decreasing first and then increasing, and an extreme point will be formed at 15%.

[0164] When the missing rate reaches the range of 20% to 50%, the error value increases with the increase of the missing rate, and a large increase occurs. This is because the increase in the number of sections with missing data in the road network begins to damage the overall spatial information of the road network, but the remaining valid sections still reach half or more, which can provide sufficient information for data completion. When the missing rate reaches more than 50%, more than half of the sections in the road network have missing data, and large areas of information are lost in the road network, causing great damage to the spatial feature information in the road network. The remaining traffic flow feature values ​​gradually begin to fail to provide sufficient spatial feature support for missing data completion.

[0165] The plotting of the average values ​​of the experimental evaluation indicators and their changing trend curves is not enough for us to conduct a more in-depth analysis of the experimental results. Fig. 9 It is a dot-line graph of experimental error under different data missing rates. Fig. 9 As shown in the figure, the average MAE of missing data completion at each time under different data missing rates is obtained, and a box plot is made based on the MAE distribution to show its maximum and minimum values, median, quartiles and third and fourth quartiles. It can be seen from the box plot that not only the average value of the evaluation index, but also the overall distribution of MAE also shows a situation of first decreasing and then increasing with the increase of data missing rate, which is consistent with the results analysis above. In addition, with the increase of data confirmation rate, the distribution range and extreme range of MAE from the fourth quartile to the third quartile are also expanding, which shows that the stability of the model completion results is also decreasing with the loss of spatial features. It is more difficult to complete some sections with serious loss of adjacent spatial information, resulting in larger error extremes.

[0166] But in general, the RNERT model achieved relatively superior results in the experiment. After pre-training, the model did not fine-tune specific tasks and expand the downstream network structure. It only relied on the pre-trained model itself to complete the missing values ​​of the single-time road network traffic flow without relying on time series features. And when the amount of missing data is in the range of 5% to 30%, the completion effect of MAPE of 6.2304% to 11.3785% can be achieved. Even if the missing amount reaches 40% or even 50%, and only half of the data is available, the model can still complete the missing values ​​with an accuracy of 12% MAPE. In addition, the RNERT model's traffic missing data completion application is more suitable for road network data, and the model input does not require multi-step time series traffic flow data. Only relying on the single-time road network node feature values ​​and the road network adjacency relationship matrix can complete the calculation through the deep extraction of spatial features.

[0167] After analyzing and evaluating the overall effect of the experiment, a certain section of the road network was selected to analyze the data completion results on it, and the performance of the model on the specific section was further evaluated. In order to facilitate the unified comparison of experimental results, when analyzing the effect of specific sections, in addition to random mask selection, a specified mask operation was performed on the specific section to control the presence of the "Wanliu Bridge-Lize Bridge" section node in the mask node of the model input data, simulating the scenario where all data of this section is missing. Fig.10a , Fig.10b , Fig.10c This is a comparison chart of the model completion effect under different data missing rates. Fig.10a , Fig.10b , Fig.10c As shown, the completion results of different experiments with data missing rates ranging from 5% to 70% in the road network are plotted in chronological order.

[0168] Depend on Fig.10a , Fig.10b , Fig.10c It can be seen that when the missing rate is small, the completed value is very close to the true value. As the missing rate increases, the difference between the completed value and the true value gradually increases. However, under almost all missing rates, the model output value can fit the changing trend of traffic flow to a certain extent, especially when the data missing rate is small, the model output is particularly close to the node true value curve. Without the input time series, the model only extracts the road network features at a single moment to obtain the current effect, which can not only ensure the accuracy of missing value completion, but also reduce the sample requirements for data completion.

[0169] In summary, the RNERT model has achieved good results in the application of missing traffic data. It can accurately complete the missing road data by relying only on the traffic flow data of the remaining sections of the road network at a certain time. Moreover, the model has shown relatively excellent results under different data missing rates. When the missing rate is less than or even equal to 50%, the model can effectively complete the missing values ​​with a small error, which has significant application value; when the missing rate is greater than 50% or even reaches 70%, the model can still estimate and restore the trend of traffic flow changes, which has important guiding significance. At the same time, this also proves that the RNERT model does have a very powerful spatial feature extraction capability, and can be used as a pre-training model to quickly adapt to specific traffic data missing completion tasks and achieve considerable application results.

[0170] (3) Road network traffic flow time series modeling experiment

[0171] The hyperparameter that needs to be tuned experimentally is the initial value of the learning rate LR. Some of the hyperparameter tuning experimental results are shown in Table 6, in order to analyze the changing trend of the model error with LR. Fig.11This is a dot-line graph of the GRU-RNERT model hyperparameter tuning experiment results. Fig.11 As shown, the experimental results are plotted in the form of a dot-line graph for analysis and comparison.

[0172] Combined with the dot-line graph, it can be seen that when the initial value of the model learning rate is within the range of less than 0.01, the model parameters are updated in the direction of gradient descent without sufficient step length, and cannot converge at the global optimal point, and are prone to fall into the local minimum point. When the initial value of the learning rate is greater than 0.01, the model parameter adjustment step is too large, and it is also impossible to accurately converge at the optimal solution, resulting in poor final training effect of the model and increased error. In summary, this paper will select 0.01 as the initial value of the learning rate of the GRU-RNERT model to achieve the best convergence effect of the model.

[0173] Table 6 GRU-RNERT model hyperparameter tuning experimental results

[0174]

[0175] In order to conduct a more comprehensive analysis of the performance of the model in traffic flow prediction, the model performs multi-step extrapolation calculations on the input of the test set after training. The specific calculation process is to add the extrapolation result to the end of the sequence and use it as the model input for the next forward pass, and finally obtain a multi-step extrapolation sequence of the input sequence. The data collection granularity in the data set used in the experiment is 5 minutes, so each step of extrapolation calculation performed by the model is a prediction of the road network traffic flow 5 minutes later. In the experiment, the model was used to perform a maximum of 10 steps, or 60 minutes, on the input of the test set. Table 7 shows the average error index of the results of different step-length extrapolation of the GRU-RNERT and the GRU-RNERT (Att) model with the soft attention mechanism, that is, the prediction of traffic flows of different future durations.

[0176] Table 7 Average error of different prediction step lengths

[0177]

[0178]

[0179] It can be seen that the three evaluation indicators of MSE, MAE and MAPE show that the GRU-RNERT (Att) and GRU-RNERT models have excellent prediction effects, which means that from the perspective of average error, the GRU-RNERT (Att) and GRU-RNERT models have smaller errors and more accurate predictions of road network timing. After analyzing the trend of the average value of the error indicators, the model error distribution under different prediction step sizes is further calculated. Fig.12a-c is a dotted line graph of the average error of different prediction step sizes. The MAE distribution of GRU-RNERT is as follows Fig.13 As shown in the figure, it can be seen that with the increase of the prediction step, the overall distribution of the model error also shows an upward trend. This shows that in the process of adding the model output to the end of the input sequence and continuing to extrapolate the sequence, the deviation in the model output continues to accumulate, causing the error between the predicted value and the true value to continue to increase. This is also due to the gradual distortion of the spatiotemporal features in the input sequence. The model extracts the wrong feature value and predicts the traffic flow at the next moment based on it, resulting in more serious distortion of the feature. This cycle repeats, eventually causing the model output error to increase with the increase of the prediction time step. Fig.14 The box plots of the average absolute error distribution of different prediction steps of each model are shown in the figure. As shown in the figure, through the error evaluation index of the model, it can be seen that the GRU-RNERT model is indeed better than the traditional recurrent neural network in road network traffic flow prediction, and its resistance to information distortion in multi-step prediction is also more superior.

[0180] After statistically analyzing the mean and distribution of the overall error of the model output, we continued to study the traffic flow prediction performance of the model on a single road section. Fig.15a , Fig.15b , Fig.15c , Fig.15d This is a comparison chart of the results of different prediction steps for each model. Fig.15a , Fig.15b , Fig.15c , Fig.15d As shown in the figure, the predictions of GRU, GRU-RNERT and GRU-RNERT (Att) at different time steps on the "Madian Bridge-Jimen Bridge" section are plotted. It can be seen that although the model output can fit the overall trend of the real traffic flow, as the prediction step increases, the fit between the four model curves and the true value curve gradually weakens. As shown in the curve graph, when predicting the traffic flow after 30 minutes and 60 minutes, due to the accumulation of errors in multi-step predictions, the model output shows a phenomenon similar to the overall lag of the curve, but it is not difficult to see that the lag of GRU-RNERT (Att) is the weakest, followed by GRU-RNERT, and the GRU model is the most serious. Combined with the analysis and description in the previous article, this shows that GRU-RNERT (Att) and GRU-RNERT have a model structure based on RNERT spatial feature extraction. Compared with GRU, the two have a more accurate fit to the inherent laws of road network traffic flow, and can capture the inherent characteristics of the data more deeply and comprehensively from multiple dimensions of time and space, resulting in more accurate extrapolation of time series, smaller errors, and smaller error accumulation in multi-step extrapolation, achieving better performance than the GRU model.

[0181] In summary, the RNERT pre-training model has a significant effect in the application of road network traffic flow prediction. Compared with the traditional recurrent neural network, the GRU-RNERT model based on RNERT can not only remember the time information in the time series data, but also use the powerful spatial feature extraction unit to complete the extraction and aggregation of spatial information, and more comprehensively remember the intrinsic characteristics of the road network traffic flow time series data, so that it can achieve better results than the benchmark model when using the spatiotemporal features to extrapolate the sequence. In addition, GRU-RNERT can also reduce the model error by obtaining more feature information in the multi-step prediction of the road network traffic flow time series, thereby reducing the error accumulation in the multi-step prediction and resisting the interference from the wrong information in the input sequence. These excellent performances make the RNERT and GRU-RNERT models provide more model foundations for road network time series modeling.

[0182] The experiment finally showed that the RNERT model does have a strong ability to extract road network spatial features after pre-training, and can quickly adapt to road network traffic flow prediction tasks and support specific downstream traffic problems. The GRU-RNERT model proposed based on RNERT can also provide a reliable and effective model method for road network traffic flow prediction.

Claims

1. A traffic network encoding representation learning method based on mask graph attention mechanism, characterized in that: The following steps are involved: Step 1: Combine the bidirectional encoding representation model and the graph attention network to build a traffic network encoding representation model RNERT based on the masked graph attention mechanism. Use mask pre-training to train the model's general ability to extract road network spatial features to support different traffic tasks. Step 1 further includes the following sub-steps: Sub-step S11, model input, including taking the average speed of traffic flow as the model input eigenvalue, the model input is the eigenvalue matrix and adjacency matrix of the road network nodes at a certain moment; The eigenvalue matrix is ​​normalized, and the maximum and minimum normalization method is used to convert the average speed into a decimal in the interval [0,1]. The calculation method is shown in formula (1): Sub-step S12, model coding calculation, the road network coding representation model is composed of multiple coding layer Encoders, whose input is aggregated into the current node through multi-layer coding transmission to obtain the node feature matrix with the same dimension as the input and containing the road network spatial information; the node feature matrix refers to the data matrix composed of traffic flow indicators in all sections of the road network at a certain moment, the node refers to the section in the road network, and the node feature value refers to the traffic flow data on the section; the road network adjacency matrix is ​​a 0, 1 matrix, which represents the connection relationship between nodes of different sections, and the corresponding positions of two sections with connection relationship are assigned 1 in the matrix; the coding layer Encoder is composed of a multi-head graph attention layer M-GAT, a residual normalization layer A&LN and a multi-layer perception mechanism feedforward transmission layer MLP; the calculation process of the road network coding representation model includes: the l-th coding layer Encoder l The forward transmission process in is expressed as shown in equations (2)-(5): s1=M-GAT(h l-1 ,A) (2), s2=A&LN(h l-1 ,s1) (3), s3=MLP(s2) (4), <h2 style=";text-align:left;direction:ltr">h<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> =A&LN(s2,s3) (5), Among them, s1, s2∈R n×K s3∈R n×1 is the intermediate transfer variable of the encoding layer; h l-1 ∈R n×1 is the output of the previous encoding layer; h l ∈R n×1 is the output of the current coding layer; A∈R n×n is the road network adjacency matrix; K is the number of heads of the multi-head attention mechanism; M-GAT(·) is the multi-head attention spatial information aggregation; A&LN(·) is the residual normalization calculation; MLP(·) is the feedforward transfer calculation; Sub-step S13, model mask pre-training, perform random masking operation on the input, set the feature value to 0 or null at a ratio of 15%, forward propagate the masked input data, and obtain the output with the same dimension; use the loss function to calculate the error loss between the true value of the masked area and the model output value, solve the gradient, and back propagate the error until the model converges; Step 2: Use the pre-trained RNERT model to complete the missing values ​​of the traffic network at a certain moment; Step 3: Based on RNERT, a traffic network gated recurrent network GRU-RNERT is constructed. RNERT is embedded into the recurrent neural network as a pre-trained spatial feature extraction unit, and participates in the calculation of the gated unit, so that it can extract the spatiotemporal multi-dimensional features of the traffic flow in the traffic network and realize the time series modeling of the traffic flow in the road network.

2. According to claim 1, a traffic network encoding representation learning method based on mask graph attention mechanism is characterized in that: The calculation process of the multi-head graph attention layer M-GAT is expressed as shown in equations (6)-(8): Among them, e k ij is the node importance between nodes i and j in the k-th layer graph attention network; is the eigenvalue of node i and node j; is the shared weight matrix of the k-th layer graph attention network; a k ij is the importance coefficient between nodes i and j in the k-th layer graph attention network; N i is the adjacent set of node i; a∈R 2 is the concatenated vector; K is the number of heads of the multi-head attention mechanism; h' i ∈R K is the node feature value after attention calculation and concatenation; LeakyReLU(·), Softmax(·), f(·) are activation functions, and σ(·) in the model design takes the LeakyReLU(·) activation function; The forward transfer process of the residual normalization layer of the first layer in the coding layer is shown in equations (9)-(12): a=(h l-1 W A&LN +s1) (9), Where a∈R n×K is the residual value of the output model of the encoding layer input and the multi-head attention layer M-GAT; W A&LN ∈R K is the weight vector of the residual module; K is the number of multi-head attention heads; n is regarded as the number of network nodes in the previous layer, and the layer normalization is processed according to the dimension n, and the data distribution changes in this dimension; μ, σ∈R K are the mean and variance of a respectively; ε takes zero value; g and b are gain value and bias value respectively, and the effect of layer normalization is guaranteed by learning g and b; s2∈R n×K is the output of the residual normalization layer; the calculation formula of the residual normalization layer of the second layer of the coding layer is the same as that of the first layer; The calculation method of the feedforward transfer layer is shown in formula (13): Among them, W MLP1 ∈R F×K , W MLP2 ∈R F is the weight matrix of the fully connected layer; b MLP1 、b MLP2 is the bias vector; F is the number of hidden nodes in the feedforward transfer layer, which is a model hyperparameter and is optimized through experiments; σ(·) is the activation function, and LeakyReLU(·) is used in the model.

3. According to claim 1, a traffic network encoding representation learning method based on mask graph attention mechanism is characterized in that: Step 3 further includes the following sub-steps: Sub-step S31, model input, the GRU-RNERT model is a time series with a step length of L, the extracted node feature values ​​are also normalized, and the maximum and minimum normalization method is used to convert the average speed into a decimal in the interval [0,1]; Sub-step S32, model loop calculation; The model framework of the GRU-RNERT model combines the gated recurrent network and the RNERT model, extracts spatial information extraction capabilities through the RNERT unit, and embeds the model into the model loop calculation process, so that the memory unit simultaneously resets and memorizes the spatiotemporal features, and memorizes the node feature values, adjacency matrix and model output of the previous moment in the model input sequence. After the internal memory unit of the model memorizes the spatiotemporal information of the input sequence, it continues to output it as a hidden vector h until the entire sequence is input. The transfer formula is expressed as shown in formulas (14)-(15): h1=GRU-RNERT(x1,h0,A) h2=GRU-RNERT(x2,h1,A) h3=GRU-RNERT(x3,h2,A) ... h L =GRU-RNERT(x L ,h L-1 ,A) (14), and L =LeakyReLU(W o h L ) (15), Among them, x1, x2, x3 and x L ∈R n×1 is the node feature matrix in the sequence, L is the length of the input sequence; h0, h1, h2, h L-1 With h L ∈R n×1 is a hidden vector that memorizes the spatiotemporal characteristics of the traffic flow sequence, and h0 takes zero value; W o Output weight matrix for the model; A∈R n×n is the road network adjacency matrix; L ∈R n×1 is the final output of the model; GRU-RNERT(·) is the forward calculation process of the GRU-RNERT model; Sub-step S33, model time series modeling task training, GRU-RNERT is implemented based on the RNERT pre-training model, so the model training is divided into two parts: RNERT unit pre-training and GRU-RNERT specific task training; GRU-RNERT specific task training adopts time series data prediction tasks, and trains the model's road network traffic flow sequence spatiotemporal feature extraction capability by extrapolating and predicting road network time series data, wherein the RNERT unit parameters are not fixed, and its pre-training results are regarded as the initial parameter values, and all model parameters are trained and adjusted; the second step of model training adopts a supervised gradient descent algorithm, and inputs a road network eigenvalue matrix time series with a step size of L, extrapolates the feature matrix of the next moment of the sequence, and adds the model output to the end of the sequence as input for the next extrapolation calculation, and finally calculates the error function value between the model output and the real data and performs back propagation, updates the model parameters, and trains the model to extract spatial and temporal features from road network time series data.

4. According to claim 3, a traffic network coding representation learning method based on mask graph attention mechanism is characterized in that: In sub-step S32, the soft attention mechanism is introduced to calculate the final output of the model, and the final output of the GRU-RNERT model is expressed as shown in equations (16)-(17): h' L =At(h1,h2,h3,...,h L-1 .h L ) (16), and L =LeakyReLU(W o h' L ) (17), The specific calculation process of Att(·) is shown in equations (18)-(20): Among them, e kL is the importance between the hidden vectors at two moments in the sequence, calculated by performing a dot product between the two; a kL is the attention coefficient between hidden vectors. The hidden vectors are weighted and summed according to the attention coefficient, and the final weighted calculation is made on the information contained in the historical input to obtain the final output of the model.

5. According to claim 3, a traffic network coding representation learning method based on mask graph attention mechanism is characterized in that: When pre-training the traffic network encoding representation model RNERT to fit the spatial law of the road network, the mean square error is selected as the loss function, and the MSE calculation formula is expressed as shown in formula (21): Where n is the total number of samples; With y i are the model output and sample true value of the corresponding positions respectively.

6. According to claim 3, a traffic network coding representation learning method based on mask graph attention mechanism is characterized in that: The specific transmission of the GRU-RNERT model is shown in equations (22)-(27): r'=W r (x t ||h t-1 ) (22), r=Sigmoid(RNERT(r',A)) (23), z'=W z (x t ||h t-1 ) (24), z=Sigmoid(RNERT(z',A)) (25) h t '=tanh(RNERT(W h (x t ||(r⊙h t-1 )),A)) (26), h t =z⊙h t-1 +(1-z)⊙h t ' (27), Among them, W r , W z , W h is the weight matrix, which performs linear calculations with the concatenated matrix of the input and hidden vectors to change the dimension of the concatenated matrix so that it can meet the input requirements of the RNERT unit; t ∈R n×1 is the node feature matrix at time t in the input sequence, A∈R n×n is the road network adjacency matrix; r∈R n×1 To reset the gate, reset the hidden value of the previous moment; z∈R n×1 is the update gate, which determines the proportion of the previous hidden value and the current temporary hidden value in the output; r' and z'∈R n×1 It is the temporary value of resetting the gate and updating the gate; h t-1 ∈R n×1 is the hidden value of the previous moment; h t Hide the value at the current moment; h t ' is the temporary hidden value at the current moment; ⊙ is the Hadamard product, that is, the multiplication of corresponding elements of the matrix; Sigmoid(·) and tanh(·) are activation functions; RNERT(·) is the forward calculation of the RNERT unit.

Images