A systematic approach to improving traffic flow data quality

Abstract

The application provides a systematic method for improving traffic flow data quality, belongs to the technical field of traffic flow data processing, and is used for solving the technical problem of low traffic flow data repairing precision of the prior art. The method is a set of methods for improving data quality designed for different application conditions of traffic flow data, including identification of abnormal data and selection of repairing methods; abnormal data is screened out through data integrity test, variance test, combination test, threshold test and zero value test by combining online and offline traffic data, and then the data is repaired through the time and space correlation of the data; the application screens abnormal data through multiple methods, overcomes the problem of incomplete data screening in single threshold test, simultaneously uses combination test, considers the relationship between traffic flow parameters, selects appropriate data repairing methods according to the characteristics of abnormal data, improves the precision of data repairing, and thus improves the overall quality of traffic flow data.

Description

Technical Field

[0001] This invention belongs to the field of traffic flow data processing technology and relates to a systematic method for improving the quality of traffic flow data. Background Technology

[0002] With the application of road monitoring and control systems, a large amount of traffic data has been collected, especially traffic flow data, which greatly helps researchers analyze road operation. By analyzing traffic flow data, we can explore the road operation and discover road operation problems. It can be said that traffic flow data is the foundation of traffic control, management, evaluation application and research.

[0003] However, existing traffic flow data acquired by some detectors often contains much noise-contaminated or missing data. Directly applying this collected traffic flow data can lead to a mismatch between the analysis results and the actual road conditions. Due to the diversity of detection equipment, methods, and approaches, and the fact that traffic flow is a collective emergence of numerous micro-individual decisions, it possesses high complexity and nonlinearity. Furthermore, data transmission via wired or wireless networks can be contaminated or lost due to electromagnetic interference or channel instability. Clearly, the causes and processes leading to low data quality during data acquisition are numerous and complex. Problems in the collected traffic flow data can be summarized into three categories: missing data, data distortion, and data anomalies. Errors in fixed-point detection data are mainly caused by two factors: firstly, improper operation during the installation, maintenance, and data transmission of fixed-point detectors, or harsh operating environments leading to equipment malfunction and data loss; secondly, traffic flow anomalies, such as abnormally dense traffic flow or frequent lane changes within the detector's detection range, can cause the equipment to fail to correctly identify the data, resulting in distorted and abnormal data.

[0004] Existing data quality improvement methods include identifying the traffic data to be processed to obtain abnormal data, then using K-fold cross-validation to compensate for the missing data and correct the erroneous data; and removing redundant data based on data similarity.

[0005] There are also methods to determine whether the data is continuous over time. If it is continuous, it is considered continuous data; if it is not, it is considered random data. For continuous data, there are also methods to determine if it follows a pattern. If it does, it is considered periodic data, and time correlation is used to correct outliers. If it does not follow a pattern, it is considered non-periodic data, and normal data from the same group as the outliers is used to correct them. Finally, there are methods to determine if the size of outliers in random data exceeds a preset threshold. If it does, the data in that group is discarded; if it does not exceed the threshold, the outliers are set to zero.

[0006] Existing solutions almost universally employ algorithms to compensate for missing data, correct erroneous data, and remove redundant data based on data similarity. Alternatively, they may identify patterns in the data's temporal continuity to supplement the data, or use thresholds to control data quality. In summary, current methods for improving traffic data quality still face the following challenges:

[0007] The characteristics of traffic data were not taken into account. Traffic flow data includes three parameters: flow rate, speed, and time occupancy. The combination of these three parameters should conform to the traffic flow pattern. If such relationships are not adequately considered, the prediction results will not conform to the actual operation pattern.

[0008] Because traffic parameters such as flow rate and vehicle speed are not independent entities, but are interconnected, using the single-parameter threshold method to independently evaluate each traffic parameter will lead to an incomplete evaluation of data quality.

[0009] Data processing suffers from lag. Some methods for improving the quality of traffic data involve classifying the collected data into categories and then using various algorithms or time and space information for repair. However, it is impossible to distinguish the data at the time of collection, which leads to a significant time consumption later on.

[0010] The methods for data repair are relatively simplistic, relying on only one type of method without considering whether a particular method is applicable to all abnormal data, resulting in low accuracy in data repair.

[0011] Based on this, in view of the data anomalies that may occur during the transmission of traffic survey data, which may lead to data quality problems and make subsequent data analysis difficult, a systematic method for improving the quality of traffic flow data is proposed. This method improves the quality of traffic flow data and enhances the accuracy of data repair to facilitate subsequent data analysis. Summary of the Invention

[0012] The purpose of this invention is to address the aforementioned problems in existing technologies by proposing a systematic method for improving the quality of traffic flow data. The technical problem this invention aims to solve is: how to improve the quality of traffic flow data.

[0013] The objective of this invention can be achieved through the following technical solutions:

[0014] A systematic method for improving the quality of traffic flow data involves online evaluation of collected traffic flow data to identify missing, abnormal, and correct data. Then, offline evaluation is used to calculate the missing rate, error rate, and daily average traffic volume change rate. Finally, the traffic flow data is repaired to improve its quality. The specific steps are as follows:

[0015] Step 1: Data Integrity Verification: Check the integrity of the raw data online. Since the data acquisition device uploads a data packet every 5 minutes, the number of data packets that each site should have per day is fixed at 288. By counting the number of data packets for each site per day and comparing it with the number of data packets that each site should have per day, the difference between the two can be used to determine the number of missing data packets.

[0016] Step 2: Perform a variance test on the complete data: Check if the variances of the three parameters—speed, flow rate, and time occupancy—are zero. If they are zero, mark them as outliers. In practice, for various reasons, the value of a certain field may remain fixed. With the same sample size, the larger the variance, the greater the data volatility and instability. Therefore, starting from the second-order properties of the data, we can use variance calculation to assess the volatility of each field's data.

[0017] Step 3: Combination Test: For data whose variance test results are not zero, a combination test is performed. Based on the definition of abnormal combinations in traffic flow theory, the flow rate, speed, and time occupancy are combined and judged. Incorrect combinations are categorized into the following five types: V>0, Q=0, O=0; V=0, Q=0, O>0; V=0, Q>0, O=0; V=0, Q>0, O>0; V>0, Q=0, O>0; where V is speed, Q is flow rate, and O is time occupancy. Data with incorrect combinations are then marked as abnormal data.

[0018] Step 4: Threshold Testing: After the combined judgment, threshold testing is performed. Threshold testing includes flow rate threshold testing, speed threshold testing, time occupancy threshold testing, following percentage threshold testing, and headway threshold testing. Abnormally high values ​​can be identified by setting thresholds; when data exceeds the set threshold, it is marked as abnormal data.

[0019] 4.1 Flow Threshold Verification: Define the flow detection value Reasonable range: ;

[0020] Where T is the time interval for collecting traffic parameters. This is a correction factor, typically taken as 1.3 to 1.5. This represents the maximum capacity of the lane. Since vehicle counting is completed in a relatively short time (tens of seconds or minutes), therefore... The range is the maximum traffic capacity of the road. The product of the product with a certain time period is at least 0.

[0021] 4.2 Speed ​​Threshold Test: Define the vehicle speed detection value at the defined location. The reasonable range is: ;

[0022] in, To detect speed limits on roads, different road classifications have different speed limits. This is a correction factor, typically taken as 1.3 to 1.5.

[0023] 4.3 Time Occupancy Threshold Test:

[0024] Time occupancy rate refers to the ratio of the time a detector is occupied by vehicles to the detection time within a given period. When traffic volume is low, vehicle speeds are correspondingly high, resulting in fewer vehicles passing the detector per unit time and a low occupancy rate. As traffic volume increases, vehicle speeds decrease, and the time the detector is occupied per unit time is longer, leading to a significantly higher occupancy rate. Generally, the occupancy rate will not exceed 95%. Therefore, the threshold for time occupancy rate screening is typically set at 95%. However, due to the presence of traffic signals, high occupancy rates can easily occur when vehicles are queuing, making this detection method unsuitable for signal-controlled intersections. Considering that most detectors are located near intersections in practice, based on analysis of actual data, the final threshold for time occupancy rate is set at 100%.

[0025] 4.4, Percentage of vehicles following each other threshold test:

[0026] To investigate the threshold for the following percentage, statistical analysis was conducted on the following percentage of traffic flow every 5 minutes on national highways, provincial highways, and county roads in plains, mountainous areas, and mountainous regions, and a threshold for the following percentage was set. Analysis of the results shows that the following percentage on county roads, provincial highways, and national highways is between 0% and 100%; the results for mountainous areas are similar. Therefore, the threshold for the following percentage in the online test was first set to [0, 100%]. Subsequently, abnormal data can be identified based on the distribution characteristics of the following percentage.

[0027] 4.5. Threshold test for vehicle frontage distance;

[0028] To investigate the threshold for vehicle headway, statistical analysis was conducted on the 5-minute traffic flow and vehicle headway data for national, provincial, and county roads in plains, mountainous, and hilly areas. Based on the analysis of the collected data, a threshold for vehicle headway was established. Comparison of vehicle headway data collected from roads with different terrains revealed that the more unfavorable the terrain and the lower the traffic flow, the more dispersed the distribution of vehicle headway values.

[0029] Step 5, Zero Value Test: After the threshold test, a zero value test is performed. Traffic flow data may show all values ​​as zero. There are two possible reasons for this:

[0030] In the first category, free-flowing situations, no vehicles arrive within the specified interval.

[0031] The second category is caused by factors such as data transmission interruption or equipment failure, which result in data being incorrectly stored as a zero value.

[0032] This paper analyzes the characteristics of free-flow traffic and proposes a method to distinguish whether zero values ​​in the data represent the first type of case. In free-flow traffic, vehicle arrivals follow a Poisson distribution, and its probability density function is expressed as:

[0033]

[0034] in, Let x be the probability density function of the number of vehicles arriving within the sampling interval. Let be the average number of vehicles arriving within the sampling interval, and e be the base of the natural logarithm; the probability of a vehicle arriving within a 5-minute sampling interval is . ,Right now Solving for the problem, we get the following: If the average arrival rate of vehicles at a cross-section within a certain detection interval is less than λ in historical data, then the zero value collected within the detection interval is considered to be true zero; otherwise, it is considered false zero, and the traffic flow data with false zero is marked as abnormal data.

[0035] Step Six: Data Analysis: Based on the results of the online evaluation, the overall situation of the data is analyzed using offline evaluation methods. Based on the screening results, the missing rate, error rate, and average daily traffic flow change rate are calculated.

[0036] 6.1 Missing Rate: The percentage of data not collected by a traffic survey device within a certain time period, out of the theoretically expected amount of data collected by that device within that time period. The calculation formula is:

[0037]

[0038] in, The missing rate; The number of traffic survey devices that failed to collect data within a certain time period; This represents the amount of data that the traffic survey equipment should theoretically collect within this time frame.

[0039] 6.2 Anomaly Rate: The percentage of problematic data identified by the online quality assessment method within a certain period, out of the total amount of data actually collected within that period. The calculation formula is:

[0040]

[0041] in, For the anomaly rate; The amount of problematic data identified by the online quality assessment method within a certain time period; This represents the actual amount of data collected during that period.

[0042] 6.3 Average daily traffic flow change rate: For different evaluation periods, the average daily traffic flow change rate can be calculated using different periods. This invention proposes a calculation method with year, month, and day as evaluation periods.

[0043] 6.31. Annual average daily traffic flow change rate;

[0044] This method calculates the annual average daily equivalent traffic volume using raw data from the previous year and the current year at the same monitoring point, and finally calculates the rate of change from the previous year to the current year.

[0045]

[0046] in, This represents the annual average daily traffic flow change rate; This represents the average daily traffic flow for that year. This represents the average daily traffic flow over the previous year.

[0047] 6.32 Monthly Average Daily Traffic Flow Change Rate: This section uses the original data of the previous month and the current month to calculate the daily average equivalent traffic volume of a certain section in a certain month, and finally calculates the change rate of the current month compared with the previous month.

[0048]

[0049] in, This represents the monthly average daily traffic flow change rate; This represents the average daily traffic flow for the current month. This represents the average daily traffic flow for the previous month.

[0050] 6.33 Daily Traffic Flow Change Rate: This section uses the raw data of the previous day and the current day to calculate the daily equivalent traffic volume of a certain section in a certain month, and finally calculates the change rate of the current day compared with the previous day.

[0051]

[0052] in, The daily traffic flow change rate; This represents the daily traffic flow for that day. This represents the daily traffic flow for the previous day.

[0053] Step 7: Data Repair: Repair data related to traffic flow, speed, following percentage, time occupancy, and headway.

[0054] Since traffic flow and speed data are directly collected and are of the highest importance for traffic condition assessment, while the following percentage has no theoretical relationship with other data, the traffic flow, speed, and following percentage are corrected first. Secondly, because time occupancy is closely related to traffic flow and speed, this characteristic is utilized in the correction method to correct the time occupancy data. Similarly, after obtaining the above data, the headway can be calculated and corrected last.

[0055] 7.1 Data Repair for Traffic Issues

[0056] Repairing traffic data issues first requires analyzing the site where the problem occurred. The analysis consists of three parts: whether the site has neighboring sites; whether the data before the problem data was normal; and whether the problem data was collected on a weekday.

[0057] Next, the site and weekday information were analyzed. The former was repaired based on spatial correlation (preferred), while the latter was repaired based on time correlation.

[0058] The basis for repairing spatial correlation is as follows: determine whether it is an isolated station. If it is not an isolated station, determine whether there are adjacent lanes. If so, collect traffic data of adjacent lanes for repair. If it is an isolated station, determine whether there are upstream and downstream relationships. If so, collect traffic data of upstream and downstream for repair.

[0059] The time-related data repair is based on the following criteria: 1. Collect traffic data from previous time intervals for repair (prioritized); 2. Depending on whether it is a working day, collect traffic data from the previous day at the same time interval for repair; otherwise, collect traffic data from the previous week at the same time interval for repair.

[0060] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods, both employing linear interpolation. The spatial correlation-based repair method was given higher priority than the temporal correlation-based repair method; that is, the spatial correlation-based repair method was prioritized for data repair.

[0061] The spatial correlation-based repair method uses the following calculation formula:

[0062]

[0063] in, The repaired data results at time t. For data on adjacent lanes or upstream / downstream relationships at time t, a and b are parameters. Parameters a and b are obtained by estimating the model based on historical data from the same time period. Once parameters a and b are obtained, the data restoration result can be calculated. .

[0064] The time-dependent repair method uses the following calculation formula:

[0065]

[0066] in, This refers to data from the previous k time intervals, where k represents the total number of time intervals of data collected for the repair of problematic data. For undetermined coefficients, The value can be obtained from the estimation model based on historical data from the same period. To repair the data results.

[0067] 7.2 Speed ​​Issue Data Repair

[0068] Repairing speed-related data issues first requires analyzing the stations where the problem occurred. This analysis consists of three parts: whether the station has adjacent lanes; whether the data prior to the problematic data was normal; and whether the problematic data was collected on a weekday.

[0069] Next, lane and weekday information were analyzed. The former was repaired based on spatial correlation (preferred), while the latter was repaired based on temporal correlation.

[0070] The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, vehicle speed data of adjacent lanes are collected for repair.

[0071] The time-related data repair is based on the following criteria: 1. Collect speed data from previous time intervals for repair (prioritize this method); 2. Depending on whether it is a working day, collect traffic data from the same time interval of the previous day for repair; otherwise, collect speed data from the same time interval of the previous week for repair.

[0072] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods. Both methods employed linear interpolation, with spatial correlation-based repair methods given priority over temporal correlation-based repair methods.

[0073] 7.3 Time occupancy issue data repair

[0074] To fix the time occupancy issue, the first step is to analyze the time when the problem occurred. This analysis consists of two parts: whether the data before the problem data was normal; and whether the problem data was collected on a weekday.

[0075] Next, the weekday information was analyzed, and repairs were made based on time correlation.

[0076] The time correlation repair is based on the following: 1. Collect time occupancy data from previous time intervals for repair (priority consideration); 2. Depending on whether it is a working day, collect time occupancy data from the previous day for the same time interval for repair; otherwise, collect time occupancy data from the previous week for the same time interval for repair.

[0077] Finally, a time-dependent repair method was used, which was a linear interpolation method.

[0078] 7.4 Data Repair for Front-End Spacing Issues

[0079] Repairing the headway data issue first requires analyzing the stations where the problem occurs. The analysis consists of two parts: whether the station has adjacent lanes; and whether the data before the problematic data was normal.

[0080] Next, lane and time information are analyzed. The former is repaired based on spatial correlation (preferred), while the latter is repaired based on time correlation.

[0081] The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, the headway data of adjacent lanes is collected for repair.

[0082] The time-related data was corrected by collecting and processing the headway data from previous time intervals.

[0083] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods. Both methods employed linear interpolation, with spatial correlation-based repair methods given priority over temporal correlation-based repair methods.

[0084] 7.5. Data repair for the percentage of vehicles following each other.

[0085] To fix the issue of the percentage of vehicles following the vehicle, the first step is to analyze when the problem occurred and whether the data collected before the problem occurred was normal.

[0086] The time information was then analyzed, and the repair method based on time correlation was: to collect the percentage data of following vehicles in the previous time interval for repair.

[0087] Finally, a spatial correlation-based repair method was used, which was a linear interpolation method.

[0088] By repairing data such as traffic flow, speed, following percentage, time occupancy, and headway, the repaired data becomes the correct data.

[0089] Compared with existing technologies, this systematic method for improving traffic flow data quality has the following advantages:

[0090] 1. This invention designs a set of traffic flow data quality improvement methods for different application scenarios of traffic flow data. These methods include anomaly identification and data repair method selection. By combining online and offline traffic data analysis, and relying on data integrity checks, variance checks, combination checks, threshold checks, and zero-value checks, anomaly data is screened out. Then, the data is repaired based on its temporal and spatial correlation. This invention overcomes the problem of incomplete data screening in single threshold checks by using multiple methods to screen anomaly data. Simultaneously, the use of combination checks considers the relationships between traffic flow parameters, and selects appropriate data repair methods based on the characteristics of the anomaly data, improving the accuracy of data repair and thus effectively improving the overall quality of traffic flow data.

[0091] 2. This invention improves the quality of traffic flow data by conducting online evaluation of the collected traffic flow data, filtering out missing, abnormal, and correct data, and then calculating the missing rate, error rate, and daily average traffic volume change rate of the data through offline evaluation.

[0092] 3. This invention can perform real-time data detection, filter data during the data collection process, and then classify problematic data, saving data processing time and improving work efficiency.

[0093] 4. This invention overcomes the drawbacks of judging problematic data based on a single threshold. It uses a combination of variance test, combination test, zero value test and threshold test to screen problematic data, resulting in more accurate screening results and facilitating precise repair.

[0094] 5. Traffic flow parameters are not independent. This invention considers the relationship between traffic flow parameters and judges the rationality of the data by the consistency of traffic flow, vehicle speed and time occupancy, thereby improving the quality of data repair. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the process structure of the method of the present invention;

[0096] Figure 2This is a schematic diagram of the process structure for data repair of traffic problems in this invention;

[0097] Figure 3 This is a schematic diagram of the process structure for fixing the speed problem in this invention;

[0098] Figure 4 This is a schematic diagram of the process structure for data repair of time occupancy issues in this invention;

[0099] Figure 5 This is a schematic diagram of the process structure for repairing the vehicle frontage distance problem data in this invention;

[0100] Figure 6 This is a schematic diagram of the process structure for repairing the vehicle following percentage problem data in this invention; Detailed Implementation

[0101] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0102] The embodiments of this patent are described in detail below, with examples of these embodiments shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.

[0103] Please see Figure 1 This embodiment provides a systematic method for improving the quality of traffic flow data. It involves online evaluation of the collected traffic flow data to identify missing, abnormal, and correct data. Then, offline evaluation is used to calculate the missing rate, error rate, and daily average traffic volume change rate. Finally, the traffic flow data is repaired to improve its quality. The specific steps are as follows:

[0104] Step 1: Data Integrity Verification: Check the integrity of the raw data online. The data acquisition device uploads a data packet every 5 minutes. Each site should have 288 data packets per day. Count the number of data packets for each site per day and compare it with the number of data packets that each site should have per day. The difference between the two can be used to calculate the number of missing data packets.

[0105] Step 2: Perform a variance test on the complete data: Check if the variances of the three parameters—speed, flow rate, and time occupancy—are zero. If they are zero, mark them as outliers. In practice, for various reasons, the value of a certain field may remain constant. With the same sample size, the larger the variance, the greater the data volatility and instability. Therefore, starting from the second-order properties of the data, we can use variance calculation to assess the volatility of each field's data.

[0106] Step 3: Combination Test: For data whose variance test results are not zero, a combination test is performed. Based on the definition of abnormal combinations in traffic flow theory, the flow rate, speed, and time occupancy are combined and judged. The following five combinations are incorrect: V>0, Q=0, O=0; V=0, Q=0, O>0; V=0, Q>0, O=0; V=0, Q>0, O>0; V>0, Q=0, O>0; as shown in Table 1 below, where V is speed, Q is flow rate, and O is time occupancy. Data with incorrect combinations are then marked as abnormal data.

[0107] Table 1 Incorrect Combination Forms

[0108] Combination number Incorrect combination form 1 V>0, Q=0, O=0 2 V=0, Q=0, O>0 3 V=0, Q>0, O=0 4 V=0, Q>0, O>0 5 V>0, Q=0, O>0

[0109] Step 4, Threshold Testing: After the combined judgment, threshold testing is performed. Threshold testing includes flow rate threshold testing, speed threshold testing, time occupancy threshold testing, following percentage threshold testing, and headway threshold testing. Abnormally high values ​​are identified by setting thresholds; when data exceeds the set threshold, it is marked as abnormal data.

[0110] 4.1 Flow threshold test: Define the flow detection value The reasonable range is: ;

[0111] Where T is the time interval for collecting traffic parameters. For correction factor, The value range is 1.3 to 1.5. This represents the maximum capacity of the lane. Since vehicle counting is completed in a relatively short time (tens of seconds or minutes), therefore... The range is the maximum traffic capacity of the road. The product of the product with a certain time period is at least 0.

[0112] 4.2 Speed ​​Threshold Test: Define the vehicle speed detection value at the defined location. The reasonable range is: ;

[0113] in, To detect speed limits on roads, different road classifications have different speed limits. For correction factor, The value range is 1.3 to 1.5.

[0114] 4.3 Time Occupancy Threshold Test:

[0115] Time occupancy rate refers to the ratio of the time a detector is occupied by a vehicle to the total detection time within a given time period. In this embodiment, the threshold for time occupancy rate is set to 100%.

[0116] When traffic volume is low, vehicle speeds are correspondingly high, resulting in fewer vehicles passing the detector per unit time and a low detector occupancy rate. As traffic volume increases, vehicle speeds decrease, and the detector is occupied for a longer period per unit time, leading to a significantly higher detector occupancy rate. Generally, the occupancy rate does not exceed 95%. Therefore, the threshold for time occupancy rate filtering is typically set at 95%. However, due to the presence of traffic signals, high occupancy rates can easily occur when vehicles are queuing, making this detection method unsuitable for signal-controlled intersections. Considering that most detectors are actually located near intersections, based on analysis of actual data, the final threshold for time occupancy rate is set at 100%.

[0117] 4.4, Percentage of vehicles following each other threshold test:

[0118] To study the threshold for the percentage of vehicles following each other, statistical analysis was conducted on the percentage of vehicles following each other every 5 minutes on national highways, provincial highways, and county roads in plains, mountainous areas, and mountain ranges, and a threshold for the percentage of vehicles following each other was set.

[0119] Analysis of the results shows that the following percentage on county roads, provincial roads, and national roads is between 0% and 100%. The same applies to mountainous areas. Therefore, the following percentage threshold for online testing is first set to [0, 100%], and then abnormal data is identified based on the distribution characteristics of the following percentage.

[0120] 4.5. Threshold test for vehicle frontage distance;

[0121] To investigate the threshold for vehicle headway, statistical analysis was conducted on the 5-minute traffic flow and vehicle headway data for national, provincial, and county roads in plains, mountainous, and hilly areas. Based on the analysis of the collected data, a threshold for vehicle headway was established. Comparison of vehicle headway data collected from roads with different terrains revealed that the more unfavorable the terrain and the lower the traffic flow, the more dispersed the distribution of vehicle headway values.

[0122] For example, the maximum collected headway was 65,535 m, but the time period in which this maximum value occurred was not continuous. Statistics were compiled based on headway values ​​corresponding to traffic flows of less than 3. The vast majority of headway values ​​were less than 25,000 m. Because the more unfavorable the terrain and the lower the traffic flow, the more dispersed the headway values ​​became. Based on the data distribution, the recommended headway threshold values ​​are shown in Table 2 below:

[0123] Table 2 Thresholds for vehicle frontage

[0124] Plains mountain mountainous areas national highway 2000m 25000m 25000m provincial highway 2000m 25000m 25000m County road 2000m 25000m 25000m

[0125] Step 5, Zero Value Test: After the threshold test, a zero value test is performed. Traffic flow data may show all values ​​as zero. There are two possible reasons for this:

[0126] In the first category, free-flowing situations, no vehicles arrive within the specified interval.

[0127] The second category is caused by factors such as data transmission interruption or equipment failure, which result in data being incorrectly stored as a zero value.

[0128] This paper analyzes the characteristics of free-flow traffic and proposes a method to distinguish whether zero values ​​in the data represent the first type of case. In free-flow traffic, vehicle arrivals follow a Poisson distribution, and its probability density function is expressed as:

[0129]

[0130] in, Let x be the probability density function of the number of vehicles arriving within the sampling interval. Let be the average number of vehicles arriving within the sampling interval, and e be the base of the natural logarithm; the probability of a vehicle arriving within a 5-minute sampling interval is . ,Right now Solving for this, we get... If the average arrival rate of vehicles at a cross-section within a certain detection interval is less than λ in historical data, then the zero value collected within the detection interval is considered to be true zero; otherwise, it is considered false zero, and the traffic flow data with false zero is marked as abnormal data.

[0131] Step Six: Data Analysis: Based on the results of the online evaluation, the overall situation of the data is analyzed using offline evaluation methods. Based on the screening results, the missing rate, error rate, and average daily traffic flow change rate are calculated.

[0132] 6.1 Missing Rate: The percentage of data not collected by a traffic survey device within a certain time period, out of the theoretically expected amount of data collected by that device within that time period. The calculation formula is:

[0133]

[0134] in, The missing rate; The number of traffic survey devices that failed to collect data within a certain time period; This represents the amount of data that the traffic survey equipment should theoretically collect within this time frame.

[0135] 6.2 Anomaly Rate: The percentage of problematic data identified by the online quality assessment method within a certain period, out of the total amount of data actually collected within that period. The calculation formula is:

[0136]

[0137] in, For the anomaly rate; The amount of problematic data identified by the online quality assessment method within a certain time period; This represents the actual amount of data collected during that period.

[0138] 6.3 Average daily traffic flow change rate: For different evaluation periods, the average daily traffic flow change rate can be calculated using different periods. This invention proposes a calculation method with year, month, and day as evaluation periods.

[0139] 6.31. Annual average daily traffic flow change rate;

[0140] This method calculates the annual average daily equivalent traffic volume using raw data from the previous year and the current year at the same monitoring point, and finally calculates the rate of change from the previous year to the current year.

[0141]

[0142] in, This represents the annual average daily traffic flow change rate; This represents the average daily traffic flow for that year. This represents the average daily traffic flow over the previous year.

[0143] 6.32 Monthly Average Daily Traffic Flow Change Rate: This section uses the original data of the previous month and the current month to calculate the daily average equivalent traffic volume of a certain section in a certain month, and finally calculates the change rate of the current month compared with the previous month.

[0144]

[0145] in, This represents the monthly average daily traffic flow change rate; This represents the average daily traffic flow for the current month. This represents the average daily traffic flow for the previous month.

[0146] 6.33 Daily Traffic Flow Change Rate: This section uses the raw data of the previous day and the current day to calculate the daily equivalent traffic volume of a certain section in a certain month, and finally calculates the change rate of the current day compared with the previous day.

[0147]

[0148] in, The daily traffic flow change rate; This represents the daily traffic flow for that day. This represents the daily traffic flow for the previous day.

[0149] Step 7: Data Repair: Repair data such as traffic flow, speed, following percentage, time occupancy, and headway in sequence.

[0150] Since traffic flow and speed data are directly collected and are of the highest importance for traffic condition assessment, while the following percentage has no theoretical relationship with other data, traffic flow, speed, and following percentage are corrected first. Secondly, because time occupancy is closely related to traffic flow and speed, this characteristic is also utilized in the correction method to correct the time occupancy data. Similarly, after obtaining the above data, the headway can be calculated and corrected finally.

[0151] 7.1 Data Repair for Traffic Issues

[0152] Repairing traffic data issues first requires analyzing the site where the problem occurred. The analysis consists of three parts: whether the site has neighboring sites; whether the data before the problem data was normal; and whether the problem data was collected on a weekday.

[0153] Next, the site and weekday information were analyzed. The former was repaired based on spatial correlation (preferred), while the latter was repaired based on time correlation.

[0154] The basis for repairing spatial correlation is as follows: determine whether it is an isolated station. If it is not an isolated station, determine whether there are adjacent lanes. If so, collect traffic data of adjacent lanes for repair. If it is an isolated station, determine whether there are upstream and downstream relationships. If so, collect traffic data of upstream and downstream for repair.

[0155] The time-related data repair is based on the following criteria: 1. Collect traffic data from previous time intervals for repair (prioritized); 2. Depending on whether it is a working day, collect traffic data from the previous day at the same time interval for repair; otherwise, collect traffic data from the previous week at the same time interval for repair.

[0156] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods, both employing linear interpolation. The spatial correlation-based repair method was given higher priority than the temporal correlation-based repair method; that is, the spatial correlation-based repair method was prioritized for data repair.

[0157] 7.2 Speed ​​Issue Data Repair

[0158] Repairing speed-related data issues first requires analyzing the stations where the problem occurred. This analysis consists of three parts: whether the station has adjacent lanes; whether the data prior to the problematic data was normal; and whether the problematic data was collected on a weekday.

[0159] Next, lane and weekday information were analyzed. The former was repaired based on spatial correlation (preferred), while the latter was repaired based on temporal correlation.

[0160] The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, vehicle speed data of adjacent lanes are collected for repair.

[0161] The time-related data repair is based on the following criteria: 1. Collect speed data from previous time intervals for repair (prioritize this method); 2. Depending on whether it is a working day, collect traffic data from the same time interval of the previous day for repair; otherwise, collect speed data from the same time interval of the previous week for repair.

[0162] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods. Both methods employed linear interpolation, with spatial correlation-based repair methods given priority over temporal correlation-based repair methods.

[0163] 7.3 Time occupancy issue data repair

[0164] To fix the time occupancy issue, the first step is to analyze the time when the problem occurred. This analysis consists of two parts: whether the data before the problem data was normal; and whether the problem data was collected on a weekday.

[0165] Next, the weekday information was analyzed, and repairs were made based on time correlation.

[0166] The time correlation repair is based on the following: 1. Collect time occupancy data from previous time intervals for repair (priority consideration); 2. Depending on whether it is a working day, collect time occupancy data from the previous day for the same time interval for repair; otherwise, collect time occupancy data from the previous week for the same time interval for repair.

[0167] Finally, a time-dependent repair method was used, which was a linear interpolation method.

[0168] 7.4 Data Repair for Front-End Spacing Issues

[0169] Repairing the headway data issue first requires analyzing the stations where the problem occurs. The analysis consists of two parts: whether the station has adjacent lanes; and whether the data before the problematic data was normal.

[0170] Next, lane and time information are analyzed. The former is repaired based on spatial correlation (preferred), while the latter is repaired based on time correlation.

[0171] The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, the headway data of adjacent lanes is collected for repair.

[0172] The time-related data was corrected by collecting and processing the headway data from previous time intervals.

[0173] Ultimately, the data was repaired using both spatial correlation-based and temporal correlation-based repair methods. Both methods employed linear interpolation, with spatial correlation-based repair methods given priority over temporal correlation-based repair methods.

[0174] 7.5. Data repair for the percentage of vehicles following each other.

[0175] To fix the issue of the percentage of vehicles following the vehicle, the first step is to analyze when the problem occurred and whether the data collected before the problem occurred was normal.

[0176] The time information was then analyzed, and the repair method based on time correlation was: to collect the percentage data of following vehicles in the previous time interval for repair.

[0177] Finally, a spatial correlation-based repair method was used, which was a linear interpolation method.

[0178] The above-mentioned repair methods based on spatial correlation and those based on temporal correlation are as follows:

[0179] The spatial correlation-based repair method uses the following calculation formula:

[0180]

[0181] in, The repaired data results at time t. For data on adjacent lanes or upstream / downstream relationships at time t, a and b are parameters. Parameters a and b are obtained by estimating the model based on historical data from the same time period. Once parameters a and b are obtained, the data restoration result can be calculated. .

[0182] The time-dependent repair method uses the following calculation formula:

[0183]

[0184] in, This refers to data from the previous k time intervals, where k represents the total number of time intervals of data collected for the repair of problematic data. For undetermined coefficients, The value can be obtained from the estimation model based on historical data from the same period. To repair the data results.

[0185] By repairing data such as traffic flow, speed, following percentage, time occupancy, and headway, the repaired data becomes the correct data.

[0186] The following specific case demonstrates the effectiveness of the systematic method for improving traffic flow data quality disclosed in this embodiment in processing traffic flow data.

[0187] We analyzed one week's data from 86 stations in Beijing for a specific year. These 86 stations covered a total of 248 lanes. Of these, 15 stations were located on national highways, 34 on provincial highways, and 37 on county highways.

[0188] Data integrity check: The daily upload volume is 288, and the ideal data volume for a week should be 7*248*288=499968 records. The original data contains 424142 records; the missing data volume is 75826 records, accounting for 15.17%, that is, the missing rate is 15.17%.

[0189] A variance test was performed on the complete data: 12,307 outlier data points were identified through the variance test, accounting for 2.9%.

[0190] Combined test: By checking whether the combination of traffic flow and speed for each vehicle type is reasonable, 6 abnormal data points were identified, accounting for 0.0012%.

[0191] By analyzing the combination of traffic flow, average speed, and time occupancy for all vehicle types, 30,171 abnormal data entries were identified, accounting for 6.03%.

[0192] Threshold verification: Through threshold verification of traffic flow, speed, time occupancy, headway, and following percentage, 2160 abnormal data entries were identified, accounting for 0.43%.

[0193] Zero value test: By checking whether the traffic, speed and time occupancy rates are all 0, 2806 abnormal data entries were identified, accounting for 0.70%.

[0194] The specific filtering results are shown in Table 3 below: Table 3 Filtering Results

[0195]

[0196] In summary, 424,142 data entries were valid, and 376,692 were correct, representing 88.81% of the valid data. In anomaly detection, erroneous data can be labeled differently based on various error types. This experiment demonstrates that current anomaly detection methods can effectively identify anomalous data.

[0197] Data repair: The abnormal and missing data identified above are repaired using interpolation methods. There are three types of interpolation methods used in this study: linear interpolation, cubic spline interpolation, and third-order Hermite interpolation.

[0198] The statistical indicators of the repair results are shown in Table 4 below.

[0199] Table 4 Statistical Indicators of Repair Results

[0200]

[0201] In Table 4, RMSE represents the root mean square error, MAPE represents the mean absolute percentage error, and VAPE represents the variance of the mean absolute percentage error. Table 4 shows that among the RMSE statistical indicators, the three interpolation methods are most effective at correcting time occupancy, linear and pchip perform similarly, and the correction effect for front-end spacing is the worst. Among the mean absolute error indicators, speed correction is the best, and the VAPE indicator also reflects this.

[0202] Flow: Among the three interpolation repair methods, linear and pchip have better repair effects than spline;

[0203] Speed: The three interpolation repair methods are not significantly different, but linear interpolation has the best repair effect;

[0204] (3) Time occupancy: Among the three interpolation repair methods, linear and pchip have better repair effects than spline;

[0205] (4) Following percentage: Among the three interpolation repair methods, linear and pchip have better repair effects than spline;

[0206] Linear repairs are the most effective.

[0207] (5) Front spacing: Among the three interpolation repair methods, linear interpolation has the best repair effect;

[0208] In summary, among the three repair methods, the linear interpolation method used in this invention after data processing achieves the best results and can effectively improve the quality of traffic flow data processing.

[0209] The preferred embodiments of this patent have been described in detail above. However, this patent is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this patent.

Claims

1. A systematic method for improving the quality of traffic flow data, characterized in that, The specific steps of this method are as follows: Step 1: Data Integrity Verification: Check the integrity of the raw data online. The data acquisition device uploads a data packet every 5 minutes. Each site should have 288 data packets per day. Count the number of data packets for each site per day and compare it with the number of data packets that each site should have per day. Calculate the difference between the two to get the number of missing data packets. Step 2: Perform a variance test on the complete data: check whether the variances of the three parameters, speed, flow rate, and time occupancy, are 0. If they are 0, mark them as outlier data. Step 3: Combination Test: For data whose variance test results are not zero, a combination test is performed. Based on the definition of abnormal combinations in traffic flow theory, the flow rate, speed, and time occupancy are combined and judged. Incorrect combinations are categorized into the following five types: V>0, Q=0, O=0; V=0, Q=0, O>0; V=0, Q>0, O=0; V=0, Q>0, O>0; V>0, Q=0, O>0; where V is speed, Q is flow rate, and O is time occupancy. Data with incorrect combinations are then marked as outliers. Step 4, Threshold Inspection: Threshold inspection is divided into flow rate threshold inspection, speed threshold inspection, time occupancy threshold inspection, following percentage threshold inspection, and headway threshold inspection; abnormally high values ​​are identified by setting thresholds. When data exceeds the set threshold, it is marked as abnormal data. Step 5, Zero Value Test: After the threshold test, a zero value test is performed. Traffic flow data may show all values ​​as zero. There are two possible reasons for this: In the first category, under free-flow conditions, no vehicles arrive within the interval. The second type is where data is incorrectly stored as a zero value due to data transmission interruption or equipment failure. This paper analyzes the characteristics of free-flow traffic and proposes a method to distinguish whether zero values ​​in the data represent the first type of case. In free-flow traffic, vehicle arrivals follow a Poisson distribution, and its probability density function is expressed as: in, Let x be the probability density function of the number of vehicles arriving within the sampling interval. Let be the average number of vehicles arriving within the sampling interval, and e be the base of the natural logarithm; the probability of a vehicle arriving within a 5-minute sampling interval is . ,Right now Solving for this, we get... If the average arrival rate of vehicles at a cross section within a certain detection interval is less than λ in historical data, then the zero value collected within the detection interval is considered to be true zero; otherwise, it is considered false zero, and the traffic flow data with false zero is marked as abnormal data. Step Six: Data Analysis: Based on the results of the online evaluation, the overall situation of the data is analyzed using offline evaluation methods. Based on the screening results, the missing rate, error rate, and daily average traffic flow change rate are calculated. Step 7: Data Repair: Repair the problematic data including traffic flow, speed, following percentage, time occupancy, and headway in sequence; S7.1, Data repair for traffic issues; Repairing traffic data issues first requires analyzing the site where the problem occurred. This analysis consists of three parts: whether the site has any neighboring sites; whether the data before the problem occurred was normal; and whether the problem data was collected on a weekday. Next, the site and weekday information were analyzed. The former was repaired based on spatial correlation, while the latter was repaired based on temporal correlation. The basis for repairing spatial correlation is as follows: determine whether it is an isolated site. If it is not an isolated site, determine whether there are adjacent lanes. If there are, collect traffic data of adjacent lanes for repair. If it is an isolated site, determine whether there are upstream and downstream relationships. If there are, collect traffic data of upstream and downstream for repair. The time-related data repair is based on the following:

1. Collect traffic data from previous time intervals for repair; 2. Depending on whether it is a working day, collect traffic data from the previous day at the same time interval for repair, otherwise collect traffic data from the previous week at the same time interval for repair. Finally, the repair was carried out using a spatial correlation-based repair method and a temporal correlation-based repair method, both of which employed linear interpolation. S7.2, Speed ​​issue data repair; Repairing speed-related data first requires analyzing the stations where the problem occurred. This analysis consists of three parts: whether there are adjacent lanes at the station; whether the data before the problem occurred was normal; and whether the problem data was collected on a weekday. Next, lane and weekday information were analyzed. The former was repaired based on spatial correlation, while the latter was repaired based on temporal correlation. The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, vehicle speed data of adjacent lanes are collected for repair. The time-related data repair is based on the following:

1. Collect speed data from previous time intervals for repair; 2. Depending on whether it is a working day, collect traffic data from the same time interval of the previous day for repair, otherwise collect speed data from the same time interval of the previous week for repair. Finally, the repair was carried out using a spatial correlation-based repair method and a temporal correlation-based repair method, both of which employed linear interpolation. S7.3, Time Occupancy Issue Data Repair; To fix the time occupancy issue, the first step is to analyze when the problem occurred. This analysis consists of two parts: whether the data before the problem data was normal; and whether the problem data was collected on a weekday. Next, the weekday information was analyzed, and repairs were made based on time correlation. The time correlation repair is based on the following:

1. Collect time occupancy data from previous time intervals for repair; 2. Depending on whether it is a workday, collect time occupancy data from the previous day for the same time interval for repair; otherwise, collect time occupancy data from the previous week for the same time interval for repair. Finally, a time-dependent repair method was used, which was a linear interpolation method. S7.4, Data fix for the front-end spacing issue; Repairing the headway issue data first requires analyzing the stations where the problem occurs. The analysis consists of two parts: whether there are adjacent lanes at the station; and whether the data before the problematic data was normal. Next, lane and time information were analyzed. The former was repaired based on spatial correlation, and the latter was repaired based on temporal correlation. The basis for repairing spatial correlation is: whether there are adjacent lanes. If so, the headway data of adjacent lanes are collected for repair. The repair of time-related data is based on collecting and repairing the headway data from previous time intervals. Finally, the repair was carried out using a spatial correlation-based repair method and a temporal correlation-based repair method, both of which employed linear interpolation. S7.5, data fix for the following percentage issue; To fix the following percentage issue data, the first step is to analyze when the problem occurred and whether the data collected before the problem occurred was normal. Next, the time information was analyzed, and the repair based on time correlation was carried out by collecting the percentage data of following vehicles in the previous time intervals. Finally, a spatial correlation-based repair method was used, which was a linear interpolation method. By fixing issues with traffic flow, speed, following percentage, time occupancy, and headway, the corrected data becomes accurate.

2. The method for systematically improving traffic flow data quality according to claim 1, characterized in that, In step four, the threshold test method is as follows: S4.1 Flow Threshold Test: Define the flow detection value The reasonable range is: ; Where T is the time interval for collecting traffic parameters. For correction factor, This represents the maximum traffic capacity of the lane. S4.2 Speed ​​Threshold Test: Define the vehicle speed detection value at the location. The reasonable range is: ; in, To detect speed limits on roads, different road classifications have different speed limits. This is a correction factor; S4.3, Time Occupancy Threshold Test: Time occupancy rate refers to the ratio of the time a detector is occupied by a vehicle to the total detection time within a given time period. The threshold for time occupancy rate is set to 100%. S4.4, Following vehicle percentage threshold test: The percentage of vehicles following each other every 5 minutes was statistically analyzed for national highways, provincial highways, and county roads in plains, mountainous areas, and mountain ranges. A threshold for the percentage of vehicles following each other was set. First, the threshold for the percentage of vehicles following each other in online testing was set to [0, 100%]. Then, abnormal data was identified based on the distribution characteristics of the percentage of vehicles following each other. S4.5, Threshold test for vehicle frontage distance; Statistical analysis was conducted on the headway of traffic flow every 5 minutes on national highways, provincial highways, and county roads in plains, mountainous areas, and mountainous regions. By analyzing the collected data, a threshold for headway was set.

3. The method for systematically improving traffic flow data quality according to claim 2, characterized in that, The threshold for the distance between vehicles on national highways, provincial highways, and county roads in plain areas is set at 2000m, while the threshold for the distance between vehicles on national highways, provincial highways, and county roads in mountainous and hilly areas is set at 25000m.

4. A method for systematically improving the quality of traffic flow data according to claim 2, characterized in that, During the flow threshold verification process The value range is 1.3~1.5; during the speed threshold test, The value range is 1.3 to 1.

5.

5. A method for systematically improving the quality of traffic flow data according to claim 1, characterized in that, In step six, the missing rate and outlier rate are calculated as follows: Missing rate: The percentage of data not collected by a traffic survey device within a certain time period, out of the theoretically expected amount of data collected by that device within that time period. The calculation formula is: in, The missing rate; The number of traffic survey devices that failed to collect data within a certain time period; This is the amount of data that the traffic survey equipment should theoretically collect within this time frame; Anomaly rate: The percentage of problematic data identified by online quality assessment methods within a certain period, out of the total amount of data actually collected during that period. The calculation formula is: in, For the anomaly rate; The amount of problematic data identified by the online quality assessment method within a certain time period; This represents the actual amount of data collected during that period.

6. A method for systematically improving the quality of traffic flow data according to claim 1, characterized in that, In step six, when calculating the average daily traffic flow change rate, different periods are used for different evaluation periods. A calculation method with year, month, and day as evaluation periods is proposed: Calculation of the annual average daily traffic flow change rate: This method uses the original data of the previous year and the current year from the same monitoring point to calculate the annual average daily equivalent traffic volume, and finally calculates the change rate of the current year compared with the previous year. in, This represents the annual average daily traffic flow change rate; This represents the average daily traffic flow for that year. This represents the average daily traffic flow over the previous year. Calculation of the average daily traffic flow change rate: This part uses the original data of the previous month and the current month to calculate the average daily equivalent traffic flow of a certain section in a certain month, and finally calculates the change rate of the current month compared with the previous month. in, This represents the monthly average daily traffic flow change rate; This represents the average daily traffic flow for the current month. This represents the average daily traffic flow for the previous month. Daily traffic flow change rate calculation: This part uses the original data of the previous day and the current day to calculate the daily equivalent traffic volume of a certain section in a certain month, and finally calculates the change rate of the current day compared with the previous day; in, The daily traffic flow change rate; This represents the daily traffic flow for that day. This represents the daily traffic flow for the previous day.

7. A method for systematically improving the quality of traffic flow data according to claim 1, characterized in that, In step seven, the repair method based on spatial correlation and the repair method based on temporal correlation are as follows: The spatial correlation-based repair method uses the following calculation formula: in, The repaired data results at time t. For data on adjacent lanes or upstream / downstream relationships at time t, a and b are parameters. Parameters a and b are obtained by estimating the model based on historical data from the same time period. After obtaining parameters a and b, the data restoration result is calculated. ; The time-dependent repair method uses the following calculation formula: in, This refers to data from the previous k time intervals, where k represents the total number of time intervals of data collected for the repair of problematic data. For undetermined coefficients, The value is obtained from a model based on historical data from the same period. To repair the data results.

Images