Cleaning fusion system for urban traffic multi-source heterogeneous data

Abstract

The invention relates to the technical field of data processing, in particular to a cleaning fusion system for urban traffic multi-source heterogeneous data, which comprises a space-time correlation graph construction module used for creating a unique node identifier for each GPS data record, roadside camera data record and coil data record according to multi-source heterogeneous traffic data, and establishing a spatial adjacent relation edge by judging whether road segment identifiers to which multiple nodes belong are the same or not. According to the method, context awareness evaluation of data source reliability is achieved, meanwhile, when traffic time series data are processed, the data are decomposed into a low-frequency approximation coefficient representing a long-term trend and a high-frequency detail coefficient capturing a sudden change event, and the approximation coefficients of the multi-source data are fused through the obtained context awareness credibility score; meanwhile, a maximum value selection strategy is adopted for detail coefficients, key signal features caused by real traffic events can be reserved, and the fidelity of fused traffic data is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a system for cleaning and fusing multi-source heterogeneous data of urban traffic. Background Technology

[0002] The field of data processing technology refers to a set of technologies that perform a series of operations on raw data to transform it into organized, usable, and valuable information.

[0003] Current data processing technologies for handling multi-source heterogeneous traffic data typically employ preset fixed weights or simple averaging methods for fusion. A key shortcoming is that they treat the reliability of the data source as a static attribute, making it difficult to adapt to complex and changing traffic environments. For example, a GPS data source assigned a high weight based on historical data may experience signal quality degradation when entering tunnels or urban canyons with tall buildings. However, the static weighting mechanism struggles to adjust its reliability in real time, leading to erroneous data contaminating the final fusion result and potentially misclassifying congested sections as unobstructed. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a system for cleaning and fusing multi-source heterogeneous data in urban traffic.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a system for cleaning and fusing multi-source heterogeneous urban traffic data, comprising: The spatiotemporal correlation graph construction module is used to create a unique node identifier for each GPS data record, roadside camera data record, and loop data record based on multi-source heterogeneous traffic data. Then, it establishes spatial proximity relationship edges by determining whether the road segment identifiers of multiple nodes are the same, and establishes temporal continuity relationship edges by determining whether the timestamps of multiple nodes are continuous, thus building a spatiotemporal correlation graph of multi-source data. The data conflict quantification and evaluation module is used to extract node pairs connected by spatial proximity edges based on the spatiotemporal correlation map of the multi-source data, obtain road segment speed values ​​and traffic flow values, and calculate the absolute value of the numerical difference. According to the preset device type and credibility mapping table, an initial credibility value is assigned to each node to obtain conflict quantification and initial credibility set.

[0006] Preferably, the system further includes: The credibility iterative propagation module is used to, based on the multi-source data spatiotemporal correlation graph and the conflict quantification and initial credibility set, in each iteration, the node summarizes the credibility information from the neighboring nodes to form a support message and transmits it outward. After receiving the support messages from all neighboring nodes, the node adjusts the current credibility value by combining its own conflict metric with a preset penalty function and enhancement function. When the change in the credibility value of all nodes is less than a predetermined threshold in two consecutive iterations, the iteration stops and the context-aware credibility score is obtained. The wavelet domain denoising and fusion module is used to perform multi-level decomposition of the speed time series of each road segment based on the context-aware credibility score and the original traffic time series data to obtain low-frequency approximation coefficients and high-frequency detail coefficients. Noise in the detail coefficients is removed using a preset threshold. The context-aware credibility score is used as a weight to weight the approximation coefficients of multiple data sources. The coefficients with the largest amplitude among the detail coefficients are selected and merged to generate a reconstructed traffic state sequence.

[0007] Preferably, the spatiotemporal correlation map construction module includes: The traffic data nodeization submodule is used to extract the data source type, road segment ID and timestamp of each record based on multi-source heterogeneous traffic data, combine the data source type, road segment ID and timestamp to generate a hash value as a unique identifier for the node, and store the road segment speed value and traffic flow value as node attributes to obtain a traffic data node set. The spatial proximity relationship construction submodule is used to traverse all node pairs based on the traffic data node set, extract road segment identifiers for string matching, and if the match is successful, establish an undirected edge between the two nodes as a spatial proximity relationship edge to build a spatial association node graph. The temporal continuity relationship construction submodule is used to filter nodes from the same data source based on the spatial association node graph, sort the timestamps, and establish a directed edge between the two nodes as a temporal continuity relationship edge to build a multi-source data spatiotemporal association graph if the timestamp difference between two adjacent nodes is equal to the preset sampling interval.

[0008] Preferably, the data conflict quantification assessment module includes: The node conflict measurement submodule is used to traverse all spatial proximity edges in the spatiotemporal correlation graph of the multi-source data, extract the node pairs connected at both ends of the edge, obtain the road segment speed values ​​and perform subtraction, then obtain the traffic flow values ​​and perform subtraction, sum the absolute values ​​of the two differences, and generate the node conflict measurement value. The node initial credibility assignment submodule is used to read the device type attribute of each node according to the spatiotemporal correlation map of the multi-source data, then query the preset device type and credibility mapping table, match and assign an initial credibility value to each node, and obtain the node initial credibility sequence. The conflict credibility integration submodule is used to store the unique identifier, associated conflict metric, and corresponding initial credibility value of each node in the graph based on the conflict metric between nodes and the initial credibility sequence of the nodes, so as to obtain the conflict quantification and initial credibility set.

[0009] Preferably, the credibility iterative propagation module includes: The support message passing submodule is used to, based on the multi-source data spatiotemporal correlation graph and the conflict quantization and initial confidence set, in each iteration, read the current confidence value of all neighboring nodes, multiply it by the weight of the associated edge and sum it to form an aggregated support message, which is then broadcast to all neighboring nodes to generate inter-node support messages. The node credibility update submodule is used to obtain the updated node credibility value by receiving the inter-node support messages from all neighbors and attenuating the messages according to its own conflict metric value, and weighting the attenuated message value with its own current credibility value.

[0010] Preferably, the credibility iterative propagation module further includes: The iterative convergence determination submodule is used to store the updated node credibility value generated in this iteration based on the updated node credibility value, and calculate the absolute value by subtracting the node credibility value stored in the previous iteration one by one. When the sum of all absolute values ​​is less than a predetermined convergence threshold, the iteration process is stopped and the context-aware credibility score is obtained.

[0011] Preferably, the wavelet domain denoising and fusion module includes: The time-series data wavelet decomposition submodule is used to select the road segment speed time series based on the original traffic time-series data, set the decomposition level to 3, perform iterative high-pass and low-pass filtering, separate the low-frequency approximation coefficients representing the trend and the high-frequency detail coefficients representing the details, and obtain the decomposed wavelet coefficient set. The multi-scale coefficient denoising submodule is used to set independent noise thresholds for high-frequency detail coefficients based on the decomposed wavelet coefficient set, iterate through each value in each group of coefficients, and set the absolute value to zero if it is less than the corresponding threshold, and retain the original value if it is greater than the threshold, so as to obtain the denoised detail coefficients.

[0012] Preferably, the wavelet domain denoising and fusion module further includes: The weighted inverse transform reconstruction submodule is used to perform a weighted average of the low-frequency approximation coefficients from different data sources according to the context-aware credibility score, based on the denoised detail coefficients, the context-aware credibility score, and the low-frequency approximation coefficients in the decomposed wavelet coefficient set. Then, it selects the coefficient with the largest amplitude from the denoised detail coefficients of different data sources point by point, combines new coefficients, and performs inverse filtering and upsampling to generate a reconstructed traffic state sequence.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, multi-source heterogeneous traffic data records are constructed into a spatiotemporal correlation map with spatial proximity and temporal continuity. The differences between nodes in road segment speed and traffic flow are quantified as a conflict metric. Through an iterative message passing mechanism, the credibility of each data point is dynamically adjusted and propagated based on the consistency support of other data points in its spatiotemporal neighborhood. This ensures that data sources that exhibit abnormal behavior in specific scenarios, even if they have a good historical track record, will have their current credibility objectively reduced, thus achieving a context-aware assessment of data source reliability. Furthermore, when processing traffic time-series data, the data is decomposed into low-frequency approximation coefficients representing long-term trends and high-frequency detail coefficients capturing abrupt events. The obtained context-aware credibility scores are used to fuse the approximation coefficients of multi-source data. At the same time, a maximum value selection strategy is adopted for the detail coefficients to ensure that key signal features caused by real traffic events are retained while filtering out high-frequency noise, thereby improving the fidelity of the fused traffic data. Attached Figure Description

[0014] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figure 1 This invention provides a technical solution: a system for cleaning and fusing multi-source heterogeneous urban traffic data, comprising: The spatiotemporal correlation graph construction module is used to create a unique node identifier for each GPS data record, roadside camera data record, and loop data record based on multi-source heterogeneous traffic data. Then, it establishes spatial proximity relationship edges by determining whether the road segment identifiers of multiple nodes are the same, and establishes temporal continuity relationship edges by determining whether the timestamps of multiple nodes are continuous, thus building a spatiotemporal correlation graph of multi-source data. The data conflict quantification and assessment module is used to extract node pairs connected by spatial proximity edges based on the spatiotemporal correlation map of multi-source data, obtain road segment speed values ​​and traffic flow values, and calculate the absolute value of the numerical difference. According to the preset device type and credibility mapping table, each node is assigned an initial credibility value to obtain conflict quantification and initial credibility set. The credibility iterative propagation module is used to summarize the credibility information from neighboring nodes in each iteration based on the spatiotemporal correlation graph of multi-source data, conflict quantification and initial credibility set, to form a support message and transmit it outward. After receiving the support messages from all neighboring nodes, the node adjusts the current credibility value by combining its own conflict metric with a preset penalty function and enhancement function. When the change in credibility value of all nodes is less than a predetermined threshold in two consecutive iterations, the iteration stops and the context-aware credibility score is obtained. The wavelet domain denoising and fusion module is used to perform multi-level decomposition of the speed time series of each road segment based on the context-aware credibility score and the original traffic time series data to obtain low-frequency approximation coefficients and high-frequency detail coefficients. It uses a preset threshold to remove noise from the detail coefficients, uses the context-aware credibility score as a weight to weight the approximation coefficients from multiple data sources, and selects the coefficient with the largest amplitude among the detail coefficients for merging to generate a reconstructed traffic state sequence.

[0017] The spatiotemporal correlation graph construction module includes: The traffic data nodeization submodule is used to extract the data source type, road segment ID and timestamp of each record based on multi-source heterogeneous traffic data, combine the data source type, road segment ID and timestamp to generate a hash value as a unique identifier for the node, and store the road segment speed value and traffic flow value as node attributes to obtain a traffic data node set. The spatial proximity relationship construction submodule is used to traverse all node pairs based on the traffic data node set, extract road segment identifiers for string matching, and if the match is successful, establish an undirected edge between the two nodes as a spatial proximity relationship edge to build a spatial association node graph. The temporal continuity relationship construction submodule is used to filter nodes from the same data source based on the spatial association node graph, sort the timestamps, and establish a directed edge between the two nodes as a temporal continuity relationship edge to build a multi-source data spatiotemporal association graph if the timestamp difference between two adjacent nodes is equal to the preset sampling interval.

[0018] Specifically, based on multi-source heterogeneous traffic data, the received GPS data records, roadside camera data records, and loop detector data records are first standardized and preprocessed. The raw messages from different devices are parsed. For example, the timestamp, latitude, longitude, and speed fields in the GPRMC statement are extracted from the NMEA-0183 format messages uploaded by GPS devices; the road segment number, statistical period, traffic flow, and average speed fields contained in the XML files transmitted by roadside cameras via FTP are extracted; and the detector ID, occupancy time, and headway information in the binary stream sent by the inductive loop detector via TCP protocol are parsed. All of these are uniformly converted into an internal data structure containing data source type, road segment ID, and timestamp. Then, for each processed data record, the data source type string (e.g., "GPS_Device_123"), the road segment ID string (e.g., "Road_Seg_45"), and the formatted data ("YYYY-MM-DD") are formatted as follows: The timestamp string "HH:MM:SS" is concatenated to form a long string, such as "GPS_Device_123_Road_Seg_45_2023-10-27 10:00:00". Then, the SHA-256 hash algorithm is applied to this concatenated long string to generate a 64-bit hexadecimal string as the unique node identifier of the data record in the map. At the same time, the road segment speed value and traffic flow value parsed from the original data are stored as the node attributes. If some data sources, such as some GPS data, only provide speed and no flow information, their traffic flow attribute value is set to a specific null value, such as -1. Finally, the node identifiers and attribute sets formed after processing all data records are summarized to obtain the traffic data node set.

[0019] To efficiently construct spatial relationships based on the traffic data node set, a hash map is first created with road segment identifiers as keys and node lists as values. Each node in the traffic data node set is traversed, its road segment identifier attribute is read, and the node is appended to the corresponding key's node list in the hash map. After traversal, each key-value pair in the hash map represents the set of all data nodes located on the same road segment. Then, each node list in the hash map is traversed. In each list, for any two different nodes, such as node A and node B, since their road segment identifiers are the same, an undirected edge is established between them. This edge is marked as a spatial proximity edge, and its weight can be initially set to 1, representing that they are spatially equivalent observations. This process is achieved by combining and pairing nodes within each list. For example, for a list containing N nodes, a hash map will be created... There are undirected edges, among which This operation connects all data points collected on the same physical road segment in a graph structure, regardless of the device from which these data points come or when they are collected. After processing the node list of all road segments, these nodes and the newly created spatial proximity edges together form a graph structure, establishing a spatially related node graph.

[0020] Based on the spatial association node graph, to introduce temporal association, all nodes in the graph are first grouped according to their data source type. A hash mapping table is created with the combination of data source type and road segment ID as the key and the node list as the value. For example, the key is "Camera_01-Road_Seg_45", and the value is all data nodes generated by this camera in this road segment. After completing the grouping of all nodes, each node list in this hash mapping table is traversed, and the nodes in the list are sorted in ascending order according to their timestamp attribute to obtain a temporally ordered node sequence. Then, this sorted node sequence is traversed sequentially. For any two adjacent nodes in the sequence... For nodes i and i+1, calculate their timestamp difference and compare it with a preset sampling interval. This preset sampling interval is determined based on the inherent properties of the data source device. For example, the sampling interval of a ground loop is usually fixed at 30 seconds, while the statistical interval of a roadside camera may be set to 60 seconds. These interval values ​​are stored in a device configuration file. During the judgment, a tolerance, such as 1 second, is introduced to tolerate the slight delay that may occur in data transmission or processing. If the absolute value of the timestamp difference between the two nodes is less than the tolerance, i.e., |timestamp(i+1) - timestamp(i) - preset interval| < 1 second, then it is determined that the two data points are continuous in time. Thus, a directed edge from i to i+1 is established between the i-th node and the i+1-th node as a temporal continuity edge. After traversing and processing all groups, a temporal continuity edge is added to the original spatial association node graph, establishing a multi-source data spatiotemporal association graph.

[0021] The data conflict quantitative assessment module includes: The node conflict measurement submodule is used to traverse all spatial proximity edges in the spatiotemporal correlation graph of multi-source data, extract the node pairs connected at both ends of the edge, obtain the road segment speed values ​​and perform subtraction, then obtain the traffic flow values ​​and perform subtraction, and sum the absolute values ​​of the two differences to generate the node conflict measurement value. The node initial credibility assignment submodule is used to read the device type attribute of each node based on the spatiotemporal correlation graph of multi-source data, then query the preset device type and credibility mapping table, match and assign an initial credibility value to each node, and obtain the node initial credibility sequence. The conflict credibility integration submodule is used to store the unique identifier, associated conflict metric, and corresponding initial credibility value of each node in the graph based on the conflict metric between nodes and the initial credibility sequence of nodes, so as to obtain the conflict quantification and initial credibility set.

[0022] Specifically, based on the spatiotemporal correlation graph of multi-source data, the conflict metric calculation process is initiated. First, all edges in the graph are traversed, and all edges marked as spatial proximity edges are selected. For each selected spatial proximity edge, the two connected node pairs are extracted, such as node A and node B. Subsequently, the road segment speed values ​​(speed_A and speed_B) and traffic flow values ​​(flow_A and flow_B) are obtained from the attributes of node A and node B, respectively. Before calculation, the speed and flow data are normalized to eliminate dimensional differences. The normalization method uses min-max normalization, specifically: normalized_value = (value - min) / (max - ... The normalization process involves calculating the speed difference (min) and flow rate (max), where min and max values ​​are reasonable ranges for speed and flow determined based on historical statistical data for the road segment. For example, the speed range for a city's main road is preset to 0 to 80 km / h, and the flow rate range is 0 to 2500 vehicles / hour. For instance, node A has a speed of 50 km / h and a flow rate of 1200 vehicles / hour, while node B has a speed of 60 km / h and a flow rate of 1000 vehicles / hour. After normalization, norm_speed_A = 50 / 80 = 0.625, norm_flow_A = 1200 / 2500 = 0.48, norm_speed_B = 60 / 80 = 0.75, and norm_flow_B = 1000 / 2500 = 0.4. Then, the absolute value of the normalized speed difference is calculated as diff_speed = |0.625 - 0.75| = 0.125, and the absolute value of the normalized flow difference is diff_flow = |0.48|. -0.4| = 0.08. The absolute values ​​of these two differences are then summed using a weighted average. The weights w_speed and w_flow are preset based on traffic engineering experience. For example, on highways, speed is more critical, so w_speed = 0.6 and w_flow = 0.4 can be set. On congested urban roads, flow rate is more important, so w_speed = 0.4 and w_flow = 0.6 can be set. If w_speed = 0.5 and w_flow = 0.5 are chosen, then the conflict metric value is 0.5. 0.125 + 0.5 0.08 = 0.0625 + 0.04 = 0.1025. This calculation result is used as the conflict metric between the two nodes and stored as an attribute of the edge connecting them. After traversing all spatial proximity edges, the conflict metric between nodes is generated.

[0023] Based on the spatiotemporal correlation map of multi-source data, each node is assigned an initial confidence level reflecting the inherent reliability of its associated equipment. This process relies on a pre-defined mapping table between equipment type and confidence level. The construction of this mapping table is based on long-term evaluation of historical data from various traffic data collection devices. The specific construction process is as follows: At least three months of historical data are collected, and several road sections simultaneously deploying multiple types of equipment are selected as reference points. At these reference points, ground truth data of traffic conditions is obtained through high-precision video analysis or manual verification. Then, for each equipment type, such as "high-rise DSLR camera," "microwave radar detector," and "floating car GPS," the mean absolute percentage error (MAPE) between its measured value and the ground truth is calculated. A conversion function is then used to map the error to a confidence level, for example, confidence level = 1 - MAPE. If the MAPE exceeds an upper limit (e.g., 0.5), the confidence level is set to a minimum value (e.g., 0.5). This method yields a mapping relationship. An example mapping table is as follows: {"High-rise DSLR camera": 0.95, "Microwave radar detector": 0.90, "Inductive loop": 0.85,} After obtaining this mapping table, traverse each node in the graph, read the device type attribute of each node, use this attribute as the key to query the corresponding confidence value in the mapping table, and store the queried value as the "initial confidence" attribute of the node. For example, a node from "inductive loop" will be assigned an initial confidence value of 0.85. After completing the assignment operation for all nodes, obtain the initial confidence sequence of the nodes.

[0024] Based on the inter-node conflict metrics and the initial credibility sequence of nodes, data integration is performed to construct a structured dataset that facilitates subsequent iterative computation. Specifically, each node in the multi-source spatiotemporal correlation graph is traversed, and a data record entry is created for each node. This entry first contains the node's unique identifier, i.e., the string previously generated by hashing. Second, the entry stores the node's initial credibility value, which is directly read from the node's attributes. Finally, all conflict information related to this node is integrated. For this, all spatial proximity edges connected to the current node need to be found. For each such edge, the inter-node conflict metrics stored in its attributes, as well as the unique identifiers of the neighboring nodes connected to the other end of the edge, are extracted. This information is organized into a conflict list, where each element contains the identifier of a neighboring node and its corresponding conflict metric. For example, for node N, its data entry might be structured as: {node_id: 'hash_N', initial_credibility: 0.9, conflicts: [{neighbor_id:'hash_M1', conflict: 0.1025}, {neighbor_id: 'hash_M2', conflict: 0.2540}]}, This process concentrates the node credibility information and edge conflict information that were originally scattered in the graph structure into a view with nodes as the core, and establishes a complete file for each node containing its initial state and interaction with its neighbors. By gathering all these structured data entries of all nodes together, the conflict quantification and initial credibility set are obtained.

[0025] The credibility iterative propagation module includes: The support message passing submodule is used to read the current confidence values ​​of all neighboring nodes in each iteration based on the spatiotemporal correlation graph of multi-source data, conflict quantification and initial confidence set, multiply them by the weights of the associated edges and sum them to form an aggregated support message, which is then broadcast to all neighboring nodes to generate inter-node support messages. The node credibility update submodule is used to calculate the updated node credibility value based on the inter-node support messages and conflict quantization and the initial credibility set. Each node receives inter-node support messages from all its neighbors and attenuates the messages according to its own conflict metric. The attenuated message value is then weighted and calculated with its own current credibility value to obtain the updated node credibility value. The iterative convergence determination submodule is used to store the updated node credibility value generated in this iteration based on the updated node credibility value, and calculate the absolute value by subtracting the node credibility value stored in the previous iteration one by one. When the sum of all absolute values ​​is less than the predetermined convergence threshold, the iteration process is stopped and the context-aware credibility score is obtained.

[0026] Specifically, based on the spatiotemporal correlation graph of multi-source data and conflict quantization and initial confidence set, in each iteration, the current confidence value of all nodes is first initialized to its initial confidence value. Then, for any node in the graph, such as node i, the following operations are performed in parallel: traverse the set of all neighboring nodes of node i, and for each neighboring node j, read its current confidence value at the end of the previous iteration or in its initial state. Simultaneously, obtain the weight of the edge connecting node i and node j. This weight is set according to the type of edge. For spatially adjacent edges, the weight is determined by the conflict metric between them. The calculation shows that the specific calculation method is a decreasing function, for example... Parameter 2 is the decay coefficient, determined by optimizing the iterative convergence speed on the validation dataset. This setting results in a smaller weight for the influence between nodes with greater conflicts. For time-continuous edges, since they represent strong temporal correlations of the same device, the weight is set to a high fixed value, such as 0.95. Then, the current confidence value is obtained from all neighbors j. With the corresponding edge weight Multiply them and sum all the products. The formula is: ,in This summation represents the set of all neighboring nodes of node i. This is the aggregate support message generated by node i. This message is then logically broadcast to all neighboring nodes of node i. In the actual implementation, the message value is temporarily stored so that all neighboring nodes can read it in the next stage of credibility update. After the aggregate support message of all nodes is calculated, the inter-node support message is generated.

[0027] Based on the support messages between nodes, conflict quantification, and the initial confidence set, in the second phase of the iteration, each node updates its own confidence value in parallel. Specifically, for any node i, it collects the support messages broadcast to it by all its neighboring nodes j in the previous phase. And based on the conflict metric between node i and its neighbor j Each received message undergoes attenuation processing, which is implemented using a penalty function and an attenuation factor. The calculation method is as follows ,in The function is a hyperbolic tangent function with a range of (-1, 1), ensuring that the decay factor smoothly decreases from 1 as the conflict level increases from 0. Then, the average of all decayed support messages is calculated as the overall support from neighbors. ,in This step involves calculating the number of neighboring nodes, summarizing their overall opinions on the current node. Next, this overall support score will be... Compared with the current confidence value of node i itself A weighted calculation is performed, while introducing its initial confidence value. As a fixed trust anchor, to prevent the trust value from drifting too far during iterations, the updated trust value... The calculation formula is: ,in , , These are weighting coefficients, and the sum of the three is 1. These coefficients are set based on experience and adjusted accordingly; for example, they can be set to... , , This indicates that the current state has the highest weight, the initial trust provides the basic correction, and the neighbor support provides the external correction. Finally, the calculated... Boundary processing is performed to ensure that its value always remains within the range [0, 1]. After completing the credibility update of all nodes, the updated node credibility value is obtained.

[0028] Based on the updated node confidence values, a convergence check is performed after each iteration. First, before the iteration begins, the initial confidence values ​​of all nodes are copied and stored as the confidence values ​​from the previous iteration, denoted as . After the k-th iteration, a set of updated node confidence values ​​is obtained, denoted as . Next, traverse all N nodes in the graph, and for each node i, calculate the absolute value of the change in its confidence value during this iteration, i.e. Then, the absolute values ​​of this change across all nodes are summed to obtain the total change in global credibility. This total is compared to a predetermined convergence threshold, which is dynamically set based on the size of the graph and the required accuracy. Specifically, the threshold is calculated as: Threshold = Total Number of Nodes ,in It is a very small positive number, representing the average tolerance for changes in the credibility of each node. For example, if we set... For a graph containing 50,000 nodes, the predetermined convergence threshold is... When calculated When the confidence value is less than 0.5, it is considered that the confidence values ​​of all nodes have stabilized and the iteration process has converged, at which point the iteration stops. Furthermore, to prevent non-convergence due to improper parameter settings or data characteristics, a maximum iteration limit is set, for example, 200 iterations. If the iteration count reaches this limit and the convergence condition is still not met, the iteration is forcibly stopped. After the iteration stops, the updated node confidence values ​​calculated in the last round are used. As the final evaluation result, a context-aware credibility score is obtained.

[0029] The wavelet domain denoising and fusion module includes: The time-series data wavelet decomposition submodule is used to select the road segment speed time series based on the original traffic time-series data, set the decomposition level to 3, perform iterative high-pass and low-pass filtering, separate the low-frequency approximation coefficients representing the trend and the high-frequency detail coefficients representing the details, and obtain the decomposed wavelet coefficient set. The multi-scale coefficient denoising submodule is used to set independent noise thresholds for high-frequency detail coefficients based on the decomposed wavelet coefficient set. It iterates through each value in each group of coefficients, and if the absolute value is less than the corresponding threshold, it is set to zero; if it is greater, the original value is retained, thus obtaining the denoised detail coefficients. The weighted inverse transform reconstruction submodule is used to perform a weighted average of the low-frequency approximation coefficients from different data sources according to the context-aware credibility score, based on the denoised detail coefficients, the context-aware credibility score, and the low-frequency approximation coefficients in the decomposed wavelet coefficient set. Then, it selects the coefficient with the largest amplitude from the denoised detail coefficients of different data sources point by point, combines the new coefficients, and performs inverse filtering and upsampling to generate the reconstructed traffic state sequence.

[0030] Specifically, based on the original traffic time-series data, the first step is to filter out all relevant road segment speed time series from multi-source data for a specific road segment. For example, for road segment A, there might be a 1-minute interval speed series from roadside cameras, a 30-second interval speed series from inductive loop detectors, and a 5-minute interval speed series aggregated from floating car GPS data. Before decomposition, these sequences with different sampling frequencies need to be resampled to a common time reference using methods such as linear interpolation, for example, to a 1-minute interval. Then, a specific mother wavelet function is selected, such as Daubechies, which has good time-frequency localization characteristics. 4. Using the db4 wavelet, each velocity time series with a unified time base is independently decomposed into multiple wavelet levels, with a decomposition level of 3. The decomposition process is as follows: In the first level, the original velocity series is simultaneously passed through a high-pass filter and a low-pass filter corresponding to the db4 wavelet. The filtered result is then downsampled at intervals to obtain the high-frequency detail coefficients cD1 and low-frequency approximation coefficients cA1 of the first level. In the second level, the low-frequency approximation coefficients cA1 obtained in the first level are used as a new input sequence, and the above high-pass, low-pass filtering and downsampling process is repeated to obtain the high-frequency detail coefficients cD2 and low-frequency approximation coefficients cA2 of the second level. In the third level, the low-frequency approximation coefficients cA2 of the second level are used as input to obtain the high-frequency detail coefficients cD3 and low-frequency approximation coefficients cA3 of the third level. Thus, an original velocity series is decomposed into a set of coefficients {cA3, cD3, cD2, cD1}, where cA3 represents the most significant trend change in the velocity series, and cD3, cD2, ... cD1 captures the details of velocity fluctuations at different time scales. After completing this process on the velocity sequences of all data sources, it obtains the decomposed wavelet coefficient set.

[0031] Based on the decomposed wavelet coefficient set, the high-frequency coefficients cD1, cD2, and cD3, representing signal details, are denoised, while the low-frequency approximation coefficient cA3, representing signal trends, remains unchanged. An independent noise threshold is set for each level of detail coefficients. The threshold is calculated using a general thresholding method based on the median absolute deviation. The specific steps are as follows: For a detail coefficient sequence at a certain level, such as cD1, first calculate the median absolute deviation MAD of the sequence. Then, estimate the noise standard deviation using the formula σ = MAD / 0.6745. Finally, apply the general thresholding formula: threshold T = σ sqrt(2 The denoising threshold for this layer is calculated using log(L), where L is the length of the coefficient sequence for that layer. For example, if the length of the first-layer detail coefficient cD1 sequence from a data source is 512, and the calculated MAD is 2.5, then the noise standard deviation is estimated to be 2.5 / 0.6745 ≈ 3.71, and the corresponding denoising threshold is 3.71. sqrt(2 log(512)) ≈ 3.71 sqrt(12.48) ≈ 13.05. Using the same method, independent thresholds are calculated for the coefficients of cD2 and cD3 levels. After determining the thresholds for each level, each value in each group of detail coefficients is traversed, and a hard thresholding function is applied for denoising. Specifically, if the absolute value of a coefficient is less than or equal to the threshold of its corresponding level, the coefficient value is set to zero; if its absolute value is greater than the threshold, its original value is retained. After performing this denoising operation on the high-frequency detail coefficients of all data sources, the denoised detail coefficients are obtained.

[0032] Based on the denoised detail coefficients, context-aware confidence scores, and low-frequency approximation coefficients in the decomposed wavelet coefficient set, multi-source data fusion and reconstruction in the wavelet domain are performed. First, low-frequency approximation coefficients are fused. For the same road segment, the third-layer low-frequency approximation coefficient sequence cA3 from different data sources is weighted and averaged according to the context-aware confidence scores of their respective data sources. Specifically, at each time point, the fused low-frequency coefficient value is equal to the sum of the products of the low-frequency coefficient values ​​of each data source at that point and their confidence scores, divided by the sum of the confidence scores of all data sources. Second, high-frequency detail coefficients are fused. For the denoised detail coefficients of each layer (cD1, cD2, cD3), a strategy of selecting the largest amplitude is adopted for fusion. That is, at each time point, the absolute values ​​of the denoised detail coefficient values ​​of all data sources at that layer are compared, and the coefficient value with the largest absolute value is selected as the fused coefficient value at that point. This method preserves the most significant traffic state change features. After the fusion of low-frequency and high-frequency coefficients, a new set of fused wavelet coefficients {cA3_fused, cD3_fused, ...} is obtained. The process involves fusing cA2_fused and cD1_fused coefficients. Finally, an inverse wavelet transform is performed on these fused coefficients to reconstruct the time series. This process is the inverse of the decomposition: first, cA3_fused and cD3_fused are upsampled and passed through reconstruction low-pass and high-pass filters respectively, then added together to obtain the reconstructed cA2_fused. Next, cA1_fused is reconstructed from cA2_fused and cD2_fused. Finally, the final time series is reconstructed from cA1_fused and cD1_fused, generating the reconstructed traffic state sequence.

Claims

1. A system for cleaning and fusing multi-source heterogeneous urban traffic data, characterized in that, The system includes: The spatiotemporal correlation graph construction module is used to create a unique node identifier for each GPS data record, roadside camera data record, and loop data record based on multi-source heterogeneous traffic data. Then, it establishes spatial proximity relationship edges by determining whether the road segment identifiers of multiple nodes are the same, and establishes temporal continuity relationship edges by determining whether the timestamps of multiple nodes are continuous, thus building a spatiotemporal correlation graph of multi-source data. The data conflict quantification and evaluation module is used to extract node pairs connected by spatial proximity edges based on the spatiotemporal correlation map of the multi-source data, obtain road segment speed values ​​and traffic flow values, and calculate the absolute value of the numerical difference. According to the preset device type and credibility mapping table, an initial credibility value is assigned to each node to obtain conflict quantification and initial credibility set.

2. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 1, characterized in that, The system also includes: The credibility iterative propagation module is used to, based on the multi-source data spatiotemporal correlation graph and the conflict quantification and initial credibility set, in each iteration, the node summarizes the credibility information from the neighboring nodes to form a support message and transmits it outward. After receiving the support messages from all neighboring nodes, the node adjusts the current credibility value by combining its own conflict metric with a preset penalty function and enhancement function. When the change in the credibility value of all nodes is less than a predetermined threshold in two consecutive iterations, the iteration stops and the context-aware credibility score is obtained. The wavelet domain denoising and fusion module is used to perform multi-level decomposition of the speed time series of each road segment based on the context-aware credibility score and the original traffic time series data to obtain low-frequency approximation coefficients and high-frequency detail coefficients. Noise in the detail coefficients is removed using a preset threshold. The context-aware credibility score is used as a weight to weight the approximation coefficients of multiple data sources. The coefficients with the largest amplitude among the detail coefficients are selected and merged to generate a reconstructed traffic state sequence.

3. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 1, characterized in that, The spatiotemporal correlation map construction module includes: The traffic data nodeization submodule is used to extract the data source type, road segment ID and timestamp of each record based on multi-source heterogeneous traffic data, combine the data source type, road segment ID and timestamp to generate a hash value as a unique identifier for the node, and store the road segment speed value and traffic flow value as node attributes to obtain a traffic data node set. The spatial proximity relationship construction submodule is used to traverse all node pairs based on the traffic data node set, extract road segment identifiers for string matching, and if the match is successful, establish an undirected edge between the two nodes as a spatial proximity relationship edge to build a spatial association node graph. The temporal continuity relationship construction submodule is used to filter nodes from the same data source based on the spatial association node graph, sort the timestamps, and establish a directed edge between the two nodes as a temporal continuity relationship edge to build a multi-source data spatiotemporal association graph if the timestamp difference between two adjacent nodes is equal to the preset sampling interval.

4. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 1, characterized in that, The data conflict quantification assessment module includes: The node conflict measurement submodule is used to traverse all spatial proximity edges in the spatiotemporal correlation graph of the multi-source data, extract the node pairs connected at both ends of the edge, obtain the road segment speed values ​​and perform subtraction, then obtain the traffic flow values ​​and perform subtraction, sum the absolute values ​​of the two differences, and generate the node conflict measurement value. The node initial credibility assignment submodule is used to read the device type attribute of each node according to the spatiotemporal correlation map of the multi-source data, then query the preset device type and credibility mapping table, match and assign an initial credibility value to each node, and obtain the node initial credibility sequence. The conflict credibility integration submodule is used to store the unique identifier, associated conflict metric, and corresponding initial credibility value of each node in the graph based on the conflict metric between nodes and the initial credibility sequence of the nodes, so as to obtain the conflict quantification and initial credibility set.

5. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 2, characterized in that, The credibility iterative propagation module includes: The support message passing submodule is used to, based on the multi-source data spatiotemporal correlation graph and the conflict quantization and initial confidence set, in each iteration, read the current confidence value of all neighboring nodes, multiply it by the weight of the associated edge and sum it to form an aggregated support message, which is then broadcast to all neighboring nodes to generate inter-node support messages. The node credibility update submodule is used to obtain the updated node credibility value by receiving the inter-node support messages from all neighbors and attenuating the messages according to its own conflict metric value, and weighting the attenuated message value with its own current credibility value.

6. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 5, characterized in that, The credibility iterative propagation module also includes: The iterative convergence determination submodule is used to store the updated node credibility value generated in this iteration based on the updated node credibility value, and calculate the absolute value by subtracting the node credibility value stored in the previous iteration one by one. When the sum of all absolute values ​​is less than a predetermined convergence threshold, the iteration process is stopped and the context-aware credibility score is obtained.

7. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 2, characterized in that, The wavelet domain denoising and fusion module includes: The time-series data wavelet decomposition submodule is used to select the road segment speed time series based on the original traffic time-series data, set the decomposition level to 3, perform iterative high-pass and low-pass filtering, separate the low-frequency approximation coefficients representing the trend and the high-frequency detail coefficients representing the details, and obtain the decomposed wavelet coefficient set. The multi-scale coefficient denoising submodule is used to set independent noise thresholds for high-frequency detail coefficients based on the decomposed wavelet coefficient set, iterate through each value in each group of coefficients, and set the absolute value to zero if it is less than the corresponding threshold, and retain the original value if it is greater than the threshold, so as to obtain the denoised detail coefficients.

8. The system for cleaning and fusing multi-source heterogeneous urban traffic data according to claim 7, characterized in that, The wavelet domain denoising and fusion module also includes: The weighted inverse transform reconstruction submodule is used to perform a weighted average of the low-frequency approximation coefficients from different data sources according to the context-aware credibility score, based on the denoised detail coefficients, the context-aware credibility score, and the low-frequency approximation coefficients in the decomposed wavelet coefficient set. Then, it selects the coefficient with the largest amplitude from the denoised detail coefficients of different data sources point by point, combines new coefficients, and performs inverse filtering and upsampling to generate a reconstructed traffic state sequence.

Images