A method for diagnosing and restoring traffic data quality at checkpoints in urban road networks

Abstract

The present invention discloses a method for diagnosing and restoring traffic data quality at a checkpoint in a city road network, comprising the following steps: step 1, calculating the location speed data of each checkpoint section from the original vehicle passing data at the city checkpoint, and distinguishing the redundant data, missing data and abnormal data types of the location speed data quality; step 2, determining the rating level according to the location speed data in step 1, constructing a data quality evaluation index system at the road section level, and performing data quality evaluation step by step according to the spatial level of "meso-section-macro road network"; step 3, using multi-order tensors to characterize the spatiotemporal information of the location speed data of the city road network, and constructing multiple traffic spatiotemporal tensor structures to be restored; constructing an SVAE-WGANGP network structure, and using the adversarial training method of WGANGP to train the generation network and the discrimination network to restore the quality of the missing location speed data. The present invention has a complete data quality evaluation system, which can improve the accuracy of data quality recovery under different missing rates.

Description

Technical Field

[0001] The invention belongs to the field of intelligent transportation technology, and in particular to a method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network. Background Art

[0002] High-quality traffic spatiotemporal data can reflect the dynamic changes of traffic operation status and provide a rich data source for the judgment, evaluation and analysis of future traffic operation status. However, due to its own performance problems, external environmental influences and large data volume, traffic information monitoring equipment produces several types of problematic data such as damage, missing, abnormal, and redundant. The potential loss of spatiotemporal information of some problematic data affects the accuracy of the analysis results of road traffic operation status. Therefore, the use of effective algorithms to systematically diagnose, evaluate and restore the quality of incomplete original traffic spatiotemporal data has great research significance for the intelligent comprehensive congestion control of urban traffic.

[0003] At present, traffic data quality diagnosis is generally based on problem data identification and data quality evaluation, which are relatively independent and have not formed a comprehensive data quality diagnosis system. Data quality evaluation methods generally use hierarchical analysis, principal component analysis, and fuzzy comprehensive evaluation methods. The existing evaluation methods are not comprehensive and objective. At present, traffic data quality recovery mainly starts from the idea of ​​using statistical interpolation data and the idea of ​​estimating missing data based on regression results. However, these ideas have not effectively solved the problem of traffic data quality recovery under high missing rates. The idea of ​​data reconstruction based on deep generation verifies the effectiveness and efficiency of generative models in the research of traffic spatiotemporal data quality recovery. However, on the one hand, the current generative model still focuses on randomly generating new data samples from random noise distribution sampling, lacking the mining and utilization of the inherent characteristics of actual missing samples. On the other hand, the internal network structure of the traditional generative model only adopts a shallow fully connected form, which cannot comprehensively extract the explicit spatiotemporal laws of traffic data and the hidden feature information under the missing mechanism. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method for diagnosing and restoring the quality of traffic data at checkpoints in an urban road network, which is used to solve the problem that the prior art cannot solve the problem of restoring the quality of traffic data under high missing rates; the present invention comprehensively considers the results of discriminating the data type of the problem, evaluates the data quality of the "meso-road section-macro-road network" spatial level step by step, and uses the SVAE-WGANGP model to restore the quality of naturally missing traffic data according to the evaluation results. The method has a complete data quality evaluation system and can improve the accuracy of data quality recovery under different missing rates.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for diagnosing and restoring traffic data quality at a checkpoint on an urban road network comprises the following steps:

[0007] Step 1, calculate the location speed data of vehicles passing through each checkpoint section from the original vehicle passing data of the urban checkpoint, and identify the redundant data, missing data and abnormal data types of the location speed data;

[0008] Step 2: Determine the rating level based on the location speed data in step 1, build a data quality evaluation index system at the road section level, and evaluate the data quality level by level according to the spatial level of "meso-road section-macro-road network";

[0009] Step 3, using multi-order tensors to represent the spatiotemporal information of vehicle speed data at urban road network locations, and constructing multiple traffic spatiotemporal tensor structures to be restored;

[0010] The SVAE-WGANGP network structure is constructed, and the generative network and discriminative network are trained using the adversarial training method of WGANGP to restore the quality of vehicle speed data at missing locations.

[0011] Preferably, in step 1, the redundant data, missing data and abnormal data type discrimination specifically includes the following processes: calculating the location speed data passing through each checkpoint section, grouping the location speed data according to the same license plate, and then sorting the sampling time of the same license plate. If data with the same license plate and sampling time appear, it is judged as redundant and repeated data and deleted; calculating the adjacent sampling time difference within the group, if the sampling time difference is less than a set threshold, it is marked as redundant similar data and deleted;

[0012] Aggregate the non-redundant data in the remaining location speed data into a time granularity of 2 minutes / day, and define the time interval within a day as T M =[2(M-1),2M], M = 1, 2, 3, ..., 720, the actual sampling time granularity is t m , if t m Not in T M If t m The data at that time was considered as missing data and deleted;

[0013] For the non-redundant data and non-missing data in the vehicle speed data of the remaining locations, the 75% quantile Q3, the 25% quantile Q1 and the interquartile range IQR = Q3-Q1 are calculated according to the Boxplot discrimination criterion. If the data exceeds the threshold range Q1-1.5IQR or Q3+1.5IQR, it is judged as abnormal data and deleted.

[0014] Preferably, in step 2, the data quality evaluation index system is

[0015]

[0016]

[0017]

[0018] AR n =(1-RR n )×(1-MR n )×(1-VR n )

[0019] In the formula, RR n is the redundancy evaluation index, which indicates the redundancy rate of the nth section. The total amount of data that should be collected at the nth section at a certain sampling time interval; MR n is the completeness evaluation index, indicating the missing rate of the nth section. VR is the data that is not actually collected in the nth section; n is the effectiveness evaluation index, which indicates the failure rate of the nth section. is the abnormal data diagnosed by Boxplot discrimination criterion for the nth section; AR n is the availability evaluation index, which indicates the availability rate of the nth section; It is the sum of the lth repeated or similar vehicle passing data collected at the nth section.

[0020] Preferably, in step 2, the quality of the location vehicle speed data is comprehensively evaluated using EWM-TOPSIS, which specifically includes the following steps:

[0021] Step 221, normalizing the initial evaluation index value;

[0022] Step 222, using the EWM method to determine the information entropy weight of each evaluation index;

[0023] Step 223, constructing a weighted matrix according to the information entropy weight;

[0024] Step 224, using the TOPSIS evaluation method to calculate the road section to be evaluated and the optimal solution f according to the weighted matrix + 、The worst solution f - The Euclidean distance between and ;

[0025] Step 225, based on the Euclidean distance and Determine the relative closeness C of each road segment i ;

[0026] Step 226, based on the relative closeness C iDetermine the vehicle speed data quality index at regional road network locations, conduct data quality evaluation, and obtain comprehensive evaluation results.

[0027] Preferably, in step 3, a multi-order tensor is used to characterize the spatiotemporal information of vehicle speed data at a location in an urban road network, which specifically includes the following steps:

[0028] Step 311, after removing the abnormal TBSS data, a binary Mask matrix is ​​introduced to mark the position of the missing TBSS data;

[0029] Step 312, using the z-score method to normalize the TBSS data to obey the standard normal distribution, so as to achieve dimensional unification;

[0030] Step 313, obtain TBSS time series data of D road sections, when D is a single road section, representing the short-term t n ∈{t1,t2,t3,...,t T Traffic timing information and periodic time information within consecutive days or weeks The expression is:

[0031]

[0032]

[0033] In the formula, The number of times a single road segment is TBSS data at the moment;

[0034] When D is multiple road sections, it represents the short-time t n ∈{t1,t2,t3,...,t T} and consecutive P days or weeks The expression of traffic spatiotemporal information at the moment is:

[0035]

[0036]

[0037] In the formula, is the second-order short-time space-time tensor of multi-section TBSS data, is the d∈{1,2,3,...,D}th road section in the tth day of a certain day. n ∈{t1,t2,t3,...,t T TBSS data at the time; is the third-order periodic space-time tensor of multi-section TBSS data, is the TBSS data of the d∈{1,2,3,...}th D road section.

[0038] Preferably, the traffic space-time tensor structure types include the following 6 types:

[0039]

[0040] Preferably, in step 3, a SVAE-WGANGP network structure is constructed, and the generative network and the discriminative network are trained using the adversarial training method of WGANGP, which specifically includes the following steps:

[0041] Step 321, constructing the generative network (G) and adversarial network (D) structures of the SVAE-WGANGP network;

[0042] The input of the generator network is the delay matrix Global mean TBSS data at the last moment and TBSS data X with missing positions marked mark The encoding output is the missing features and traffic spatiotemporal information of TBSS data, and the decoding output is the reconstructed sample X recon ;The generated network structure consists of GRUD-CNN encoder and CNN-GRU decoder;

[0043] The discriminant network structure consists of a deep fully connected structure, with the original missing data X and the reconstructed sample X recon As input, by discriminating X recon The similarity with X gives the discrimination probability P∈(0,1).

[0044] Preferably, the GRUD-CNN encoder is formed by cascading a GRUD network and a CNN network, and the expression of the GRUD-CNN encoder is:

[0045]

[0046]

[0047] In the formula, It is the input of GRUD-CNN encoder; is the posterior distribution of GRUD-CNN encoding output; Z is the traffic spatiotemporal information and missing features sampled from the posterior distribution.

[0048] Preferably, the CNN-GRU decoder is composed of a CNN network cascaded with a GRU network, and the expression of the CNN-GRU decoder is:

[0049]

[0050]

[0051] In the formula, is the probability distribution of CNN-GRU decoding output; X recon is the reconstructed sample generated by resampling from the probability distribution.

[0052] Preferably, in step 3, the adversarial training method of WGANGP is used to train the generative network and the discriminative network to restore the quality of the vehicle speed data at the missing location, which specifically includes the following steps:

[0053] Step 331, the optimization goal of the generated network is to generate a reconstructed sample that is similar to the original data X. The loss function expression of the generated network is as follows:

[0054]

[0055] In the formula, is the KL divergence; is the posterior distribution assumed in advance; σ and μ are the parameters of Gaussian distribution; Loss recon is the L2 regularization reconstruction loss;

[0056] Step 332: The optimization goal of the discriminant network is to discriminate between the original data X and the reconstructed sample X recon The approximation between them is as follows:

[0057]

[0058] In the formula, Loss DF_D is the loss function of the Wasserstein distance between the approximate reconstructed sample distribution and the original data distribution, X' is the random interpolation sample sampled from the intersection of the original data distribution and the reconstructed sample distribution, λ is the penalty coefficient; Loss ISVAE_G Loss function that improves ISVAE’s ability to generate network deceptions;

[0059] Step 333, the overall loss function expression of the SVAE-WGANGP network obtained by synthesizing the loss functions of the generation network and the discrimination network is as follows:

[0060] Loss total =KLD+Loss recon +Loss DF_D +Loss ISVAE_G

[0061] Step 334, using back propagation Loss total The optimal reconstruction data obtained The expression for recovering the original naturally missing data X is as follows:

[0062]

[0063] Completed the restoration of vehicle speed data quality at missing locations.

[0064] Compared with the prior art, the present invention has the following beneficial technical effects:

[0065] The present invention provides a method for diagnosing and restoring traffic data quality at checkpoints in urban road networks. Starting from the actual problems existing in the original checkpoint traffic data itself, the redundant and missing data discrimination algorithm and the Boxplot abnormal data discrimination criterion are used to accurately discriminate the problem data type, and a comprehensive evaluation index system is constructed based on this, avoiding the complex selection process of data quality evaluation indicators, and being simple and easy to operate.

[0066] The present invention uses the EWM weighting method to determine the weights of each evaluation indicator, and then evaluates the TBSS data quality of the road section based on the TOPSIS evaluation method. It further combines road attributes to comprehensively evaluate the TBSS data quality of the regional road network, providing certain data value support for the quality recovery and application development of urban checkpoint traffic data.

[0067] The present invention restores the quality of naturally missing data under different missing mechanisms through a generative adversarial network. The generative adversarial network is composed of a GRUD-CNN encoder and a CNN-GRU decoder, which is used to extract traffic spatiotemporal information and missing feature information from naturally missing data. The discriminant network uses a multi-layer fully connected network and uses the WGANGP adversarial training method to discriminate the consistency between the actual data and the reconstructed sample distribution, and obtains the optimal reconstructed data sample, which effectively improves the accuracy of data quality recovery of the model under different missing mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of the method for diagnosing and restoring traffic data quality at urban road network checkpoints.

[0069] Figure 2 This is the flowchart for determining redundant and missing data in TBSS.

[0070] Figure 3 This is a TBSS abnormal data detection diagram based on the Boxplot discrimination criterion.

[0071] Figure 4a This is the redundancy rate result diagram of 30 road sections.

[0072] Figure 4b This is the missing rate result diagram of 30 road sections.

[0073] Figure 4c This is the failure rate result diagram of 30 road sections.

[0074] Figure 4d This is the availability result diagram of 30 road sections.

[0075] Figure 5 Schematic diagram of the mechanism of missing spatiotemporal data of vehicle speed at a location.

[0076] Figure 6 To generate a network structure diagram.

[0077] Figure 7 This is the structure diagram of the discriminant network.

[0078] Figure 8 This is a diagram showing the quality restoration effect of TBSS missing data for a certain road section for one day. DETAILED DESCRIPTION

[0079] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0080] The present invention provides a method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network, comprising the following steps:

[0081] S1. Calculate the vehicle speed (TBSS) at each checkpoint section from the original vehicle passing data at the city checkpoints, and identify the redundant data, missing data and abnormal data types for the quality of the global TBSS data;

[0082] S2. Taking the TBSS data of the urban road network as the research object, the data quality evaluation is carried out step by step according to the spatial hierarchy of "meso-road section-macro-road network". First, a data quality evaluation index system at the road section level is constructed to realize the TBSS data quality evaluation of the urban checkpoint section, and further the overall TBSS data quality at the road network level is comprehensively evaluated in combination with road attributes;

[0083] S3. Use multi-order tensors to represent the spatiotemporal information of urban road network TBSS data, and construct various types of traffic spatiotemporal tensors to serve as input for the SVAE-WGANGP data repair model; design the SVAE-WGANGP network structure, train the generative network and the discriminative network based on a WGANGP adversarial training method, and restore the quality of missing TBSS data.

[0084] The specific process in S1 is as follows: S11: Redundancy and data discrimination: Calculate the vehicle speed (TBSS) at each checkpoint section, and group the TBSS data according to the same license plate, and then sort the sampling time of the same license plate. If there is data with the same license plate and sampling time, it is judged as redundant and repeated data; secondly, calculate the difference in adjacent sampling time within the group. If the sampling time difference is less than the set threshold, it is marked as redundant similar data. After simplifying the redundant data, set the fixed time granularity to 2 minutes, and perform aggregation operations on the remaining TBSS non-redundant data. Based on this, the time interval within a day can be defined as T M=[2(M-1),2M], M = 1, 2, 3, ..., 720, the actual sampling time granularity is t m , if t m Not in T M If t m The data at that time is considered as missing data.

[0085] S12: Abnormal data identification: After removing the remaining missing TBSS data, the 75% quantile Q3, the 25% quantile Q1 and the interquartile range IQR = Q3-Q1 are calculated according to the Boxplot discrimination criterion. If the data exceeds the threshold range Q1-1.5IQR or Q3+1.5IQR, the data is considered abnormal.

[0086] The specific process of S2 is as follows: S21: Construct a multi-attribute evaluation index system for TBSS data quality. For the type of problem data, calculate the multi-attribute evaluation index system for TBSS data quality of the bayonet section from the aspects of redundancy evaluation, integrity evaluation, effectiveness evaluation and availability evaluation, including redundancy rate, missing rate, failure rate and availability rate. The specific expression is as follows:

[0087]

[0088] AR n =(1-RR n )×(1-MR n )×(1-VR n )

[0089] In the formula, RR n is the redundancy evaluation index, which indicates the redundancy rate of the nth section. The total amount of data that should be collected at the nth section at a certain sampling time interval; MR n is the completeness evaluation index, indicating the missing rate of the nth section. VR is the data that is not actually collected in the nth section; n is the effectiveness evaluation index, which indicates the failure rate of the nth section. is the abnormal data diagnosed by Boxplot discrimination criterion for the nth section; AR n is the availability evaluation index, which indicates the availability rate of the nth section; It is the sum of the lth repeated or similar vehicle passing data collected at the nth section.

[0090] S22: Comprehensive evaluation of TBSS data quality using EWM-TOPSIS

[0091] (1) Normalization of initial evaluation index values:

[0092]

[0093] In the formula, p is the number of road sections to be evaluated, q is the number of evaluation indicators; a ij is the jth index value of the road section i after data normalization, where i = 1, ..., p, j = 1, ..., q; Maxa j and Mina j They represent the maximum and minimum values ​​of the column where index j is located.

[0094] (2) Use the EWM method to determine the information entropy weight of each evaluation index:

[0095]

[0096] In the formula, w j is the weight of each indicator; H j is the information entropy of each indicator; C is a constant and is greater than 0, generally taken P ij is the index value a′ in the road section p to be evaluated ij Total index value characteristic weight.

[0097] (3) Construct the weight matrix F:

[0098]

[0099] The weight matrix F is used to calculate the best and worst solutions as well as the best and worst distances.

[0100] (4) Using the TOPSIS evaluation method to calculate the road section to be evaluated and the optimal solution f + 、The worst solution f - The Euclidean distance between and

[0101]

[0102]

[0103] The optimal and worst distances are obtained by calculating the optimal solution and the worst solution, and finally the relative closeness of each road section is calculated based on the two distances.

[0104] (5) Relative proximity of each road segment C i Sure:

[0105] (6) Determination of regional road network TBSS data quality index (TBSSDQI):

[0106] In the formula, A i is the proportion of the i-th checkpoint detection section; B iis the level of the i-th road section, which is generally divided into Level 1 for expressways, Level 2 for trunk roads, Level 3 for secondary roads and Level 4 for branch roads.

[0107] The specific process of S3 is as follows: S31: Traffic space-time tensor construction,

[0108] After removing the abnormal TBSS data, a binary Mask matrix is ​​introduced to mark the location of the missing TBSS data. The specific expression is as follows:

[0109]

[0110] In the formula, For each road segment d∈{1,2,3,...,D} at the tth n ∈{t1,t2,t3,...,t T} TBSS data at the time, if the TBSS data is not Nan, then Marked as 1, otherwise 0; X mark is a traffic tensor with the position information of missing elements marked.

[0111] The z-score method is used to normalize the TBSS data to obey the standard normal distribution and achieve dimensional unification. The specific expression is as follows:

[0112] Where μ is the mean of all samples and σ is the variance of all samples.

[0113] Further obtain the TBSS time series data of D sections. When D is a single section, it represents the short-term t n ∈{t1,t2,t3,...,t T Traffic timing information and periodic time information within consecutive days or weeks The expression is:

[0114]

[0115] In the formula, The number of times a single road segment is Furthermore, when D is a number of road segments, the short-term t n ∈{t1,t2,t3,...,t T} and consecutive P days or weeks The expression of traffic spatiotemporal information at the moment is:

[0116]

[0117] In the formula, is the second-order short-time space-time tensor of multi-section TBSS data, For the The road section is t times in a certain day n ∈{t1,t2,t3,...,t T TBSS data at the time; is the third-order periodic space-time tensor of multi-section TBSS data, is the TBSS data of the d∈{1,2,3,...}th D road section.

[0118] S32: Construction of the generative network and discriminative network structure of the SVAE-WGANGP network

[0119] Construct the generative network (G) and adversarial network (D) structures of the SVAE-WGANGP network. The generative network structure consists of a GRUD-CNN encoder and a CNN-GRU decoder. The input of the generative network is the time-delay matrix Global mean TBSS data at the last moment and TBSS data X with missing positions marked mark The encoding output is the missing features and traffic spatiotemporal information of TBSS data, and the decoding output is the reconstructed sample X recon The GRUD-CNN encoder is composed of a cascade of the GRUD network and the CNN network. The specific expression is as follows:

[0120]

[0121]

[0122] In the formula, It is the input of GRUD-CNN encoder; is the posterior distribution of GRUD-CNN encoding output; Z is the traffic spatiotemporal information and missing features sampled from the posterior distribution.

[0123] The CNN-GRU decoder consists of a CNN network cascaded with a GRU network. The specific expression is as follows:

[0124]

[0125]

[0126] In the formula, is the probability distribution of CNN-GRU decoding output; X recon is the reconstructed sample generated by resampling from the probability distribution.

[0127] The discriminant network structure consists of a deep fully connected structure, with the original missing data X and the reconstructed sample Xrecon As input, by discriminating X recon The similarity with X gives the discrimination probability P∈(0,1).

[0128] S33: Restoring TBSS missing data quality based on WGANGP adversarial training

[0129] The optimization goal of the generative network is to generate reconstructed samples that are similar to the original data X. The loss function expression of the generative network is as follows:

[0130]

[0131] In the formula, is the KL divergence; is the posterior distribution assumed in advance; σ and μ are the parameters of Gaussian distribution; Loss recon is the L2 regularized reconstruction loss.

[0132] The optimization goal of the discriminant network is to distinguish the original data X from the reconstructed sample X recon The approximation between them is as follows:

[0133]

[0134] In the formula, Loss DF_D is the loss function of the Wasserstein distance between the approximate reconstructed sample distribution and the original data distribution, X' is the random interpolation sample sampled from the intersection of the original data distribution and the reconstructed sample distribution, λ is the penalty coefficient; Loss ISVAE_G Loss function that improves the ability of ISVAE to generate deceptive networks.

[0135] The overall loss function expression of the SVAE-WGANGP network obtained by integrating the loss functions of the generation network and the discriminant network is as follows:

[0136] Loss total =KLD+Loss recon +Loss DF_D +Loss ISVAE_G

[0137] Finally, using back propagation Loss total The optimal reconstruction data obtained The expression for recovering the original naturally missing data X is as follows:

[0138]

[0139] Example

[0140] like Figure 1The method for diagnosing and restoring traffic data quality at a checkpoint in an urban road network shown in the figure comprises the following steps:

[0141] S1. Identification of problem data types: The original vehicle passing data at urban checkpoints contains dynamic data and static data. Among them, dynamic data includes instantaneous vehicle speed and passing time, and static data includes road section number and license plate number. Taking the spatial scope of a regional road network consisting of 30 directional road sections in Urumqi, Xinjiang as an example, a total of 475,200 checkpoint data records were collected from August 30, 2021 to September 5, 2021. Table 1 shows the original vehicle passing data records collected by some checkpoints. The vehicle speed (Traffic bayonet spot speed, TBSS) of each checkpoint section calculated from the original vehicle passing data is shown in Table 2.

[0142] Table 1 Original vehicle passing information at checkpoints on some sections of the road

[0143]

[0144]

[0145] Table 2 Vehicle speed information at checkpoints on some sections of the road

[0146]

[0147] S11: Redundant and missing data identification: TBSS data are grouped according to the same license plate, and then the sampling time of the same license plate is sorted. If data with the same license plate and sampling time appear, it is identified as redundant and repeated data; then the difference in adjacent sampling time within the group is calculated. If the sampling time difference is less than the set threshold, it is marked as redundant similar data. After simplifying the redundant data, the fixed time granularity is set to 2 minutes, and the remaining TBSS non-redundant data is aggregated. Based on this, the time interval within a day can be defined as T M =[2(M-1),2M], M = 1, 2, 3, ..., 720, the actual sampling time granularity is t m , if t m Not in T M If t m The data at that time is considered as missing data. The TBSS redundant and missing data judgment flow chart is as follows Figure 2 shown.

[0148] S12: Abnormal data identification: After removing the remaining missing TBSS data, the 75% quantile Q3, the 25% quantile Q1 and the interquartile range IQR = Q3-Q1 are calculated according to the Boxplot discrimination criteria. If the threshold range Q1-1.5IQR or Q3+1.5IQR is exceeded, the data is considered abnormal. The TBSS abnormal data detection diagram is shown in the figure below: Figure 3 shown.

[0149] S2. TBSS data quality evaluation: S21: Construct a multi-attribute evaluation index system for TBSS data quality.

[0150] According to the problem data type, the multi-attribute evaluation index system of TBSS data quality of the bayonet section is calculated from the aspects of redundancy evaluation, integrity evaluation, effectiveness evaluation and availability evaluation, including redundancy rate, missing rate, failure rate and availability rate. The specific expression is as follows:

[0151]

[0152]

[0153] In the formula, RR n is the redundancy evaluation index, which indicates the redundancy rate of the nth section. The total amount of data that should be collected at the nth section at a certain sampling time interval; MR n is the completeness evaluation index, indicating the missing rate of the nth section. VR is the data that is not actually collected in the nth section; n is the effectiveness evaluation index, which indicates the failure rate of the nth section. is the abnormal data diagnosed by Boxplot discrimination criterion for the nth section; AR n is the availability evaluation index, which indicates the availability rate of the nth section; It represents the sum of the lth repeated or similar vehicle passing data collected at the nth section, where the specific expression of f(l) is as follows:

[0154]

[0155] Where l is the number of time-space vehicle passing data sequences; Δt l is the previous sampling time series t l-1 With the next sampling time series t l The difference, Δt l =0 means the two data sequences are exactly the same, which is duplicate data, 0<Δt l <T means that the difference of adjacent sampling time series is within the appropriate time threshold and belongs to similar data, Δt l ≥T indicates non-redundant data.

[0156] Taking the road network checkpoint traffic data of a certain area collected from August 30, 2021 to September 5, 2021 in Urumqi, Xinjiang as an example, according to the calculation process of the evaluation indicators in S21, the results of each evaluation indicator are shown in Figure 4.

[0157] S22: Comprehensive evaluation of TBSS data quality using EWM-TOPSIS

[0158] (1) Normalization of initial evaluation index values:

[0159]

[0160] In the formula, p is the number of road sections to be evaluated, q is the number of evaluation indicators; a ij is the jth index value of the road section i after data normalization, where i = 1, ..., p, j = 1, ..., q; Maxa j and Mina j They represent the maximum and minimum values ​​of the column where index j is located.

[0161] (2) Use the EWM method to determine the information entropy weight of each evaluation index:

[0162]

[0163] In the formula, w j is the weight of each indicator; H j is the information entropy of each indicator; C is a constant and is greater than 0, generally taken P ij is the index value a′ of the road section to be evaluated ij Total index value characteristic weight.

[0164] (3) Construct the weight matrix F:

[0165]

[0166] (4) Using the TOPSIS evaluation method to calculate the road section to be evaluated and the optimal solution f + 、The worst solution f - The Euclidean distance between and :

[0167]

[0168]

[0169] (5) Relative proximity of each road segment C i Sure:

[0170] (6) Determination of regional road network TBSS data quality index (TBSSDQI):

[0171] In the formula, B iis the level of the i-th road section, which is generally divided into Level 1 for expressways, Level 2 for trunk roads, Level 3 for secondary roads, and Level 4 for branch roads; A i is the proportion of the i-th checkpoint detection section, and the specific expression is as follows:

[0172]

[0173] Where, L i L is the length of the checkpoint detection section; total The complete road length.

[0174] According to the calculation process of the EWM-TOPSIS comprehensive evaluation method in S22, the TBSS data quality evaluation results at the road section level and road network level are shown in Tables 3 and 4.

[0175] Table 3 Evaluation results of proximity of some road sections

[0176]

[0177] Table 4 TBSSDQI evaluation results of regional road network

[0178]

[0179] S3, TBSS data quality recovery

[0180] S31: Traffic space-time tensor construction

[0181] After removing the abnormal TBSS data, a binary Mask matrix is ​​introduced to mark the location of the missing TBSS data. The specific expression is as follows:

[0182]

[0183] In the formula, For each road segment d∈{1,2,3,...,D} at the tth n ∈{t1,t2,t3,...,t T} TBSS data at time}, if the TBSS data is not Nan, then M D×TP× Marked as 1, otherwise 0; X mark is a traffic tensor with the position information of missing elements marked.

[0184] The z-score method is used to normalize the TBSS data to obey the standard normal distribution and achieve dimensional unification. The specific expression is as follows:

[0185]

[0186] Where μ is the mean of all samples and σ is the variance of all samples.

[0187] Furthermore, the TBSS time series data of D sections are obtained. When D is a single section, the short-term t n ∈{t1,t2,t3,...,t T Traffic timing information and periodic time information within consecutive days or weeks The expression is:

[0188]

[0189] In the formula, The number of times a single road segment is Furthermore, when D is a number of road segments, the short-term t n ∈{t1,t2,t3,...,t T} and consecutive P days or weeks The expression of traffic spatiotemporal information at the moment is:

[0190]

[0191] In the formula, is the second-order short-time space-time tensor of multi-section TBSS data, For the The road section is t times in a certain day n ∈{t1,t2,t3,...,t T TBSS data at the time; is the third-order periodic space-time tensor of multi-section TBSS data, is the TBSS data of the d∈{1,2,3,...}th D road section.

[0192] The six types of traffic space-time tensor structure constructed according to the traffic space-time tensor representation in S31 are shown in Table 5.

[0193] Table 5 Traffic space-time tensor structure types

[0194]

[0195]

[0196] S32: Construction of the generative network and discriminative network structure of the SVAE-WGANGP network

[0197] In this embodiment, the structures of the generating network (G) and the adversarial network (D) of the SVAE-WGANGP network are as follows: Figure 6 , Figure 7The generated network structure consists of a GRUD-CNN encoder and a CNN-GRU decoder. The input of the generated network is the time delay matrix Global mean TBSS data at the last moment and TBSS data X with missing positions marked mark The encoding output is the missing features and traffic spatiotemporal information of TBSS data, and the decoding output is the reconstructed sample X recon The GRUD-CNN encoder is composed of a cascade of the GRUD network and the CNN network. The specific expression is as follows:

[0198]

[0199]

[0200] In the formula, , is the input of GRUD-CNN encoder; is the posterior distribution of the GRUD-CNN encoding output; Z is the traffic spatiotemporal information and missing features sampled from the posterior distribution. Among them, the GRUD network mainly captures the missing features and traffic time information of TBSS data. The specific expression is as follows:

[0201]

[0202] In the formula, r t 、z t , are reset gate, update gate and candidate hidden layer state respectively; h t is the hidden state vector at time t; σ(·) and tanh(·) are both activation functions; W r , W z , W h , U t , U z , U h 、V r 、V z 、V h 、b r 、b z 、b h are weight and bias terms respectively; X, h t-1 Attenuate the input and attenuate the hidden layer state respectively. The specific expressions are as follows:

[0203]

[0204] In the formula, is the dth road section at time The time-delay matrix; γ is the attenuation factor constraint coefficient, which is generally in the form of a negative exponential to ensure that the value is monotonically decreasing between [0,1]; is the dth road segment at the previous time t n -1 TBSS data; is the global average value of the dth road section; h t-1 is the hidden layer state vector at time t-1; W γ and b γ are weight bias terms respectively. Among them, The specific expression is as follows:

[0205]

[0206] In the formula, M D×T×P 0-1 elements in ; and They respectively indicate whether the corresponding position element in the traffic tensor exists. When For the moment With the previous moment The difference, when When Based on the previous moment of

[0207] The CNN network mainly extracts implicit correlation information in the spatial dimension. The specific expression is as follows:

[0208]

[0209]

[0210] In the formula, Is the input of CNN; and are the weight and bias term of the lth CNN layer respectively; ReLU(·) is the activation function of the CNN network; is the output of the CNN network.

[0211] The CNN-GRU decoder consists of a CNN network cascaded with a GRU network. The specific expression is as follows:

[0212]

[0213]

[0214] In the formula, is the probability distribution of CNN-GRU decoding output; X reconis the reconstructed sample generated by resampling from the probability distribution.

[0215] The discriminant network structure consists of a deep fully connected structure, with the original missing data X and the reconstructed sample X recon As input, by discriminating X recon The similarity with X gives the discrimination probability P∈(0,1).

[0216] S33: Restoring TBSS missing data quality based on WGANGP adversarial training

[0217] The optimization goal of the generative network is to generate reconstructed samples that are similar to the original data X. The loss function expression of the generative network is as follows:

[0218]

[0219] In the formula, is the KL divergence; is the posterior distribution assumed in advance; σ and μ are the parameters of Gaussian distribution; Loss recon is the L2 regularized reconstruction loss.

[0220] The optimization goal of the discriminant network is to distinguish the original data X from the reconstructed sample X recon The approximation between them is as follows:

[0221]

[0222] In the formula, Loss DF_D is the loss function of the Wasserstein distance between the approximate reconstructed sample distribution and the original data distribution, X' is the random interpolation sample sampled from the intersection of the original data distribution and the reconstructed sample distribution, λ is the penalty coefficient; Loss ISVAE_G Loss function that improves the ability of ISVAE to generate deceptive networks.

[0223] The overall loss function expression of the SVAE-WGANGP network obtained by integrating the loss functions of the generation network and the discriminant network is as follows:

[0224] Loss total =KLD+Loss recon +Loss DF_D +Loss ISVAE_G

[0225] Finally, using back propagation Loss total The optimal reconstruction data obtained The expression for recovering the original naturally missing data X is as follows:

[0226]

[0227] S34: Example calculation

[0228] (1) Experimental data selection

[0229] After removing the abnormal TBSS data, a total of 128,259 missing data points of the road network to be restored were obtained. The first 6 days were selected for training the model, and the last day was used as the test data set. Different data missing mechanisms were simulated according to a certain missing ratio, and a binary Mask matrix was introduced to mark the location of the missing TBSS data. The schematic diagram of the missing mechanism of TBSS data was obtained through visualization as shown in the figure below: Figure 5 shown.

[0230] (2) Determination of model parameters

[0231] In the generated network structure of the SVAE-WGANGP network, the number of neurons in the encoder's GRUD layer, CNN layer, and fully connected layer is set to 256, and the number of neurons in the decoder's fully connected layer, CNN layer, and GRU layer is set to 256. The convolution kernel size of the CNN is 3×3, and the number of convolution kernels is 720. The discriminant network consists of 3 fully connected layers combined with the LeakyReLU activation function structure. The reconstructed output result of the generated network is directly obtained from the fully connected layer, and the discriminant network's discriminant result is obtained by the Sigmoid activation function. Set the missing rate to be in the range of 5% to 50%, and randomly generate M under different missing mechanisms. D×T×P , we get the completely random point missing mechanism (PMCR), the random linear missing mechanism (LMR) and the random block missing mechanism (BMR).

[0232] (3) Experimental results presentation

[0233] Choose RMSE, MAE, R 2 As an evaluation indicator of the quality of model recovery of missing data, the specific expression is as follows:

[0234]

[0235]

[0236]

[0237] In the formula, x and x recon are the original data and the reconstructed data respectively. is the mean of the original data, and n is the total number of location speed data samples.

[0238] The results of the TBSS data quality evaluation indicators for 30 road sections for one day under different missing mechanisms are shown in Table 6. The TBSS data quality recovery effect of one road section for one day is shown in Figure 8 shown.

[0239] Table 6 Model performance evaluation index results

[0240]

[0241] The above are specific embodiments of the present invention, but the protection scope of the present invention should not be limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention, so the protection scope of the present invention should be based on the protection scope defined in the claims.

Claims

1. A method for diagnosing and restoring traffic data quality at a checkpoint on an urban road network, characterized in that: The following steps are included: Step 1, calculate the location speed data of vehicles passing through each checkpoint section from the original vehicle passing data of the urban checkpoint, and identify the redundant data, missing data and abnormal data types of the location speed data; Step 2: Determine the rating level based on the location speed data in step 1, build a data quality evaluation index system at the road section level, and evaluate the data quality level by level according to the spatial level of "meso-road section-macro-road network"; Step 3, using multi-order tensors to represent the spatiotemporal information of vehicle speed data at urban road network locations, and constructing multiple traffic spatiotemporal tensor structures to be restored; Construct the SVAE-WGANGP network structure, and use the WGANGP adversarial training method to train the generative network and the discriminative network to restore the quality of the vehicle speed data at the missing locations; In step 3, the SVAE-WGANGP network structure is constructed, and the WGANGP adversarial training method is used to train the generative network and the discriminative network, which specifically includes the following steps: Step 321, constructing the generative network (G) and adversarial network (D) structures of the SVAE-WGANGP network; The input of the generator network is the delay matrix Global mean TBSS data at the last moment and TBSS data X with missing positions marked mark The encoding output is the missing features and traffic spatiotemporal information of TBSS data, and the decoding output is the reconstructed sample X recon ;The generated network structure consists of GRUD-CNN encoder and CNN-GRU decoder; The discriminant network structure consists of a deep fully connected structure, with the original missing data X And reconstruct sample X recon As input, by discriminating X recon The similarity with X gives the discrimination probability P∈(0,1); The TBSS refers to: calculating the vehicle speed at each checkpoint section from the original vehicle passing data at the city checkpoint; The meso-road section-macro-road network refers to: first, constructing a data quality evaluation index system at the road section level to realize the TBSS data quality evaluation of urban checkpoint sections, and further combining road attributes to comprehensively evaluate the overall TBSS data quality at the road network level.

2. A method for diagnosing and restoring traffic data quality at a checkpoint in an urban road network according to claim 1, characterized in that: In step 1, the identification of redundant data, missing data and abnormal data types specifically includes the following processes: calculating the location speed data of each checkpoint section, grouping the location speed data according to the same license plate, and then sorting the sampling time of the same license plate. If data with the same license plate and sampling time appear, it is identified as redundant and repeated data and deleted; calculating the difference in adjacent sampling time within the group, if the sampling time difference is less than the set threshold, it is marked as redundant similar data and deleted; Aggregate the non-redundant data in the remaining location speed data into a time granularity of 2 minutes / day, and define the time interval within a day as T M =[2(M-1),2M], M = 1, 2, 3, ..., 720, the actual sampling time granularity is t m , if t m Not in T M If t m The data at that time was considered as missing data and deleted; For the non-redundant data and non-missing data in the vehicle speed data of the remaining locations, the 75% quantile Q3, the 25% quantile Q1 and the interquartile range IQR = Q3-Q1 are calculated according to the Boxplot discrimination criterion. If the data exceeds the threshold range Q1-1.5IQR or Q3+1.5IQR, it is judged as abnormal data and deleted.

3. The method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network according to claim 1, characterized in that: In step 2, the data quality evaluation index system is AR n =(1-RR n )×(1-MR n )×(1-VR n ) In the formula, RR n is the redundancy evaluation index, which indicates the redundancy rate of the nth section. The total amount of data that should be collected at the nth section according to a certain collection time interval; MR n is the completeness evaluation index, indicating the missing rate of the nth section. VR is the data that is not actually collected in the nth section; n is the effectiveness evaluation index, which indicates the failure rate of the nth section. is the abnormal data diagnosed by Boxplot discrimination criterion for the nth section; AR n is the availability evaluation index, which indicates the availability rate of the nth section; It is the sum of the lth repeated or similar vehicle passing data collected at the nth section.

4. The method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network according to claim 1, characterized in that: In step 2, EWM-TOPSIS is used to comprehensively evaluate the quality of the location speed data, which specifically includes the following steps: Step 221, normalizing the initial evaluation index value; Step 222, using the EWM method to determine the information entropy weight of each evaluation index; Step 223, constructing a weighted matrix according to the information entropy weight; Step 224, using the TOPSIS evaluation method to calculate the road section to be evaluated and the optimal solution f according to the weighted matrix + 、The worst solution f - The Euclidean distance between and Step 225, based on the Euclidean distance and Determine the relative closeness C of each road segment i ; Step 226, based on the relative closeness C i Determine the vehicle speed data quality index at regional road network locations, conduct data quality evaluation, and obtain comprehensive evaluation results.

5. The method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network according to claim 1, characterized in that: In step 3, a multi-order tensor is used to represent the spatiotemporal information of vehicle speed data at urban road network locations, which specifically includes the following steps: Step 311, after removing the abnormal TBSS data, a binary Mask matrix is ​​introduced to mark the position of the missing TBSS data; Step 312, using the z-score method to normalize the TBSS data to obey the standard normal distribution, so as to achieve dimensional unification; Step 313, obtain TBSS time series data of D road sections, when D is a single road section, representing the short-term t n ∈{t1,t2,t3,...,t T Traffic timing information and periodic time information within consecutive days or weeks The expression is: In the formula, The number of times a single road segment is TBSS data at the moment; When D is multiple road sections, it represents the short-time t n ∈{t1,t2,t3,...,t T } and consecutive P days or weeks The expression of traffic spatiotemporal information at the moment is: In the formula, is the second-order short-time space-time tensor of multi-section TBSS data, is the d∈{1,2,3,...,D}th road section in the tth day of a certain day. n ∈{t1,t2,t3,...,t T TBSS data at the moment; is the third-order periodic space-time tensor of multi-section TBSS data, is the TBSS data of the d∈{1,2,3,...,D}th road section.

6. The method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network according to claim 1, characterized in that: The traffic space-time tensor structure types include the following 6 types: Type 1: The traffic spatiotemporal tensor structure is: number of road sections*time slices, and the traffic information is for a short period of time; Type 2: The traffic spatiotemporal tensor structure is: number of road sections*time slice*working day, and number of road sections*time slice*non-working day, and the traffic information is a daily cycle; Type 3: The traffic space-time tensor structure is: number of road sections * time slice * one week, and the traffic information is a weekly cycle; Type 4: The traffic spatiotemporal tensor structure is: number of road sections*time slices, and the traffic information is short-term spatiotemporal; Type 5: The traffic space-time tensor structure is: number of road sections*time slice*working day, and number of road sections*time slice*non-working day. The traffic information is in daily cycle space-time. Type 6: The traffic space-time tensor structure is: number of road sections * time slice * one week, and the traffic information is in weekly space-time; Among them, the time slice is 720, the working day is 5, the non-working day is 2, and the week is 7.

7. The method for diagnosing and restoring traffic data quality at a checkpoint of an urban road network according to claim 1, characterized in that: The GRUD-CNN encoder is formed by cascading the GRUD network and the CNN network. The expression of the GRUD-CNN encoder is: Where X is the naturally missing TBSS data, M D×T×P is a binary Mask matrix, is the road segment d at time t n The time lag interval, The segment d at time t n-1 TBSS data, is the global average value of section d, Is the input of GRUD-CNN encoder; is the posterior distribution of GRUD-CNN encoding output; Z is the traffic spatiotemporal information and missing features sampled from the posterior distribution.

8. A method for diagnosing and restoring traffic data quality at a checkpoint in an urban road network according to claim 7, characterized in that: The CNN-GRU decoder consists of a CNN network cascaded with a GRU network. The expression of the CNN-GRU decoder is: In the formula, is the probability distribution of CNN-GRU decoding output; X recon is the reconstructed sample generated by resampling from the probability distribution.

9. A method for diagnosing and restoring traffic data quality at a checkpoint in an urban road network according to claim 8, characterized in that: In step 3, the adversarial training method of WGANGP is used to train the generative network and the discriminative network to restore the quality of the vehicle speed data at the missing locations, which specifically includes the following steps: Step 331, the optimization goal of the generated network is to generate X Approximately reconstruct the sample, the loss function expression of the generated network is as follows: In the formula, is the KL divergence; is the posterior distribution assumed in advance; σ and μ are the parameters of Gaussian distribution; Loss recon is the L2 regularization reconstruction loss; Step 332: The optimization goal of the discriminant network is to discriminate between the original data X and the reconstructed sample X recon The approximation between them is as follows: In the formula, Loss DF_D is the loss function of the Wasserstein distance between the approximate reconstructed sample distribution and the original data distribution, X′ is the random interpolation sample sampled from the intersection of the original data distribution and the reconstructed sample distribution, and is the penalty coefficient; Loss ISVAE_G A loss function to improve ISVAE's ability to generate network deception; Step 333, the overall loss function expression of the SVAE-WGANGP network obtained by synthesizing the loss functions of the generation network and the discrimination network is as follows: Loss total =KLD+Loss recon +Loss DF_D +Loss ISVAE_G Step 334, using back propagation Loss total The optimal reconstruction data obtained The expression for recovering the original naturally missing data X is as follows: Completed the restoration of vehicle speed data quality at missing locations.

Images