A multi-source data fusion data asset construction method for intelligent transportation

Abstract

The application discloses a kind of multi-source data fusion data asset construction methods for intelligent transportation, it is related to intelligent transportation data processing technical field, comprising: S1 collection multi-source heterogeneous data and pre-processing, constructs initial data asset library;S2 is based on Kalman filtering and combines road network topological constraint, completes trajectory data noise reduction and positioning calibration;S3 is based on space-time double branch attention mechanism, realizes multi-source data fusion weight dynamic self-adapting distribution and high-precision fusion;S4 completes the road network space matching of fusion data and the global space-time correlation of "vehicle-road-environment", constructs standardization data asset;The application adopts "Kalman filtering+attention mechanism" hybrid fusion architecture, reduces trajectory positioning error to 1.5m, travel time calculation error is reduced to within 5%, processing efficiency is promoted to more than 1 million / hour, and engineering landing ability is strong, can support intelligent transportation full-scene fine management and control.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation data processing technology, and in particular to a method for constructing multi-source data fusion data assets for intelligent transportation. Background Technology

[0002] With the rapid development of intelligent transportation systems, traffic data sources are becoming increasingly diversified and heterogeneous, encompassing multi-dimensional data such as floating car trajectories, ETC gantries, checkpoint capture data, road network topology, and meteorological conditions. This type of data is characterized by inconsistent spatiotemporal dimensions, strong noise interference, and a coexistence of information redundancy and complementarity. Multi-source data fusion is a core link connecting raw data processing and traffic feature mining, and a crucial foundation for building high-quality traffic data assets and supporting refined intelligent transportation management. Its core objective is to solve the problem of information complementarity and redundancy elimination among multi-source heterogeneous data, achieving the construction of data assets that are fully interconnected across "vehicles, roads, and the environment."

[0003] In existing technologies, fusion solutions for multi-source trajectory data in intelligent transportation mainly fall into two categories: The first category is a fixed-weighted fusion method, such as fusing floating car, ETC, and checkpoint data in a fixed ratio of 1:1:1. This method is simple in implementation logic, but it has extremely poor adaptability to the dynamic characteristics of traffic scenarios. When floating car data fluctuates briefly due to environmental interference, the fixed weights cannot adapt to changes in data reliability, leading to a sharp increase in the travel time calculation error of the fusion result to over 12%, which cannot support the construction of high-quality data assets. The second category is a pure deep learning fusion model, such as a pure deep learning traffic data fusion model based on Transformer. Although it can reduce the travel time calculation error to within 5%, it suffers from high computational complexity and inference latency exceeding 500ms, failing to meet the requirements of second-level real-time control in intelligent transportation. Meanwhile, lightweight improved algorithms suffer from insufficient accuracy and poor scenario generalization ability.

[0004] The existing traffic data fusion solutions disclosed in patents and literature have not solved the above-mentioned core problems. For example, some patents disclose fixed-weight fusion methods with poor scenario adaptability, and some literature proposes deep learning fusion models with insufficient real-time performance. Existing technologies generally cannot simultaneously take into account fusion accuracy, scenario dynamic adaptability and processing real-time performance. There is an irreconcilable technical contradiction between "accuracy priority" and "efficiency priority", making it difficult to build standardized, high-value intelligent transportation full-domain data assets and failing to meet the engineering application needs of refined management and control of intelligent transportation. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for constructing multi-source data fusion data assets for intelligent transportation. It adopts a hybrid fusion architecture of "Kalman filtering + attention mechanism" and follows a progressive processing logic of "noise reduction first and weighting later". While ensuring high-precision fusion, it achieves extremely low inference latency and dynamic adaptability across all scenarios, solving the technical pain point of the inability to balance accuracy and efficiency in existing technologies. Ultimately, it constructs standardized intelligent transportation data assets with full-domain association of "vehicle-road-environment".

[0006] To achieve the aforementioned objectives, the present invention employs a method for constructing multi-source data fusion data assets for intelligent transportation. Each step is executed based on a unified spatiotemporal benchmark and follows a progressive processing logic of noise reduction followed by weighting. Specifically, it includes the following core steps: S1. Collect and preprocess multi-source heterogeneous data of intelligent transportation to build an initial traffic data asset library: Collect full-volume heterogeneous data of intelligent transportation in the dimensions of vehicle operation, road infrastructure, and traffic environment. Perform preprocessing on the collected full-volume data, including data format unification, spatiotemporal benchmark alignment, outlier removal, and missing value completion, to form an initial traffic data asset library with unified spatiotemporal benchmark and standardized format.

[0007] S2. Noise Reduction and Calibration Based on Kalman Filter and Road Network Topology Constraints: A discrete linear Kalman filter model is constructed, with the vehicle's two-dimensional position, driving speed, and heading angle as state variables, and the positioning data from each data source as observation variables. Road network topology constraints are introduced, and trajectory points are matched with road network segments through a hidden Markov model to remove abnormal positioning points and correct the Kalman filter state prediction values. After filtering iterations, point-by-point noise reduction of multi-source trajectory data is completed, and a standardized trajectory dataset is output.

[0008] S3. Dynamic weighted fusion based on spatial-temporal dual-branch attention mechanism: Construct a dual-branch attention mechanism model that includes spatial and temporal attention branches. The spatial branch assigns spatial area weights based on quantitative indicators such as road segment congestion level and road network criticality. The temporal branch assigns temporal dimension weights based on traffic time period and traffic flow pattern. Using calibrated trajectory data as input, the output is the dynamic fusion weights of each data source. After normalization, the weighted fusion is completed to obtain the full vehicle trajectory dataset.

[0009] S4. Road Network Spatial Matching and Full-Domain Spatiotemporal Association: Using Geographic Information System tools, high-precision spatial matching of fused trajectory data and road network is achieved. Using road segment number and timestamp as the association key, the full-domain association of vehicle trajectory, road segment traffic status, and environmental influencing factors is completed, and a structured, reusable, and traceable standardized smart transportation data asset is constructed.

[0010] Optional step S5, Computing power adaptation and parallel acceleration optimization: Optimization is carried out from three aspects: hardware, software and algorithm. The core server is equipped with high-performance processors, large memory and high-speed SSD configuration, and the edge terminal is equipped with industrial-grade embedded devices; Python's Pandas and NumPy libraries and TensorFlow deep learning framework are used to complete data processing and model building; NVIDIA high-performance graphics cards are used to accelerate the core algorithm in parallel with CUDA to ensure real-time processing performance.

[0011] The core innovation of this invention is: Hardware selection: The core server uses Intel Xeon Gold 6430, 256GB memory, and 8TB high-speed SSD to meet the parallel processing of millions of trajectory points, while the edge uses industrial-grade embedded devices adapted to outdoor environments. Software selection: Pandas / NumPy is used for basic data processing, ArcGIS Pro is used for high-precision trajectory-road network spatial matching, and TensorFlow is used to build the fusion model algorithm framework. Hybrid architecture: It is the first to use a hybrid architecture of "Kalman filtering + spatial-temporal dual-branch attention mechanism", which follows the logic of "noise reduction first and weighting later" to achieve the optimal balance between accuracy and efficiency.

[0012] The beneficial effects of this invention are as follows: 1. This invention utilizes the noise reduction and calibration technology of "Kalman filtering + road network topology constraints" in S2. Based on conventional Kalman filtering, it introduces physical constraints of the road network, eliminates invalid abnormal positioning points, corrects the filtering prediction results, and reduces the trajectory positioning error from 3.2m to 1.5m. This effectively eliminates the problem of short-term data fluctuations caused by environmental interference and lays a solid data foundation for subsequent high-precision fusion.

[0013] 2. This invention achieves dynamic weight adaptive allocation through the S3 spatial-temporal dual-branch attention mechanism. It can adjust the fusion weight of each data source in real time according to the road network congestion status and time period characteristics. Compared with the traditional fixed weight fusion method, it reduces the travel time calculation error from 12% to less than 5%, which greatly improves the fusion accuracy and dynamic adaptability in all scenarios, and solves the core pain point of poor generalization ability of fixed weight schemes.

[0014] 3. This invention, through a progressive hybrid fusion architecture of "noise reduction first, weighting later" (S2+S3), can control the single-batch inference latency to within 200ms while ensuring a fusion error of less than 5%. Combined with the GPU parallel acceleration optimization of S5, the data fusion processing efficiency can be increased from 200,000 records / hour to 1 million records / hour. This solves the technical contradiction of "accuracy priority" and "efficiency priority" in the prior art, and fully meets the engineering application requirements of real-time management and control of intelligent transportation.

[0015] 4. This invention achieves high-precision spatial matching (98.7%) between trajectory data and road network through the high-precision spatial matching and full-domain spatiotemporal correlation technical solution of S4, and constructs standardized and reusable traffic data assets with full-dimensional correlation of "vehicle-road-environment". At the same time, the software and hardware adaptation solution of S5 can be directly deployed in existing intelligent transportation systems without large-scale modification of existing equipment, reducing the modification cost by more than 30%. It provides high-quality data support for all-scenario applications of intelligent transportation such as traffic control, emergency dispatch, road network planning, and travel services, and has a strong engineering implementation capability. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the multi-source data fusion data asset construction method for intelligent transportation described in this invention; Figure 2 This is a block diagram illustrating the principle of the Kalman filter noise reduction calibration module of the present invention. Figure 3 This is a heatmap showing the weight distribution of the dynamically weighted dual-branch attention mechanism of this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 , Figure 2 and Figure 3 As shown, this invention takes city-level intelligent transportation network as the core application scenario, while supplementing the generalization of the high-speed section small-scenario verification scheme. The specific implementation of the method described in this invention includes the following steps: Example 1: Multi-source data fusion and data asset construction for city-level intelligent transportation network Step 1: Collect and preprocess multi-source heterogeneous data for intelligent transportation, and build an initial traffic data asset library. The system collects comprehensive smart transportation data within the main urban area of ​​the target city. Vehicle operation data includes: floating car trajectory data for 120,000 taxis and ride-hailing vehicles (sampling frequency 1Hz); vehicle passage data from 32 ETC gantries citywide (including vehicle license plate number, passage time, and gantry location information); and capture data from 216 road checkpoints citywide (including passage time, license plate number, and location information). Road infrastructure data includes the main urban area's road network topology data (including road segment number, direction, length, number of lanes, speed limit, and other attribute data). Traffic environment data includes rainfall and visibility data from 12 meteorological monitoring stations citywide, as well as traffic event data such as road construction and traffic accidents.

[0019] The collected full data is preprocessed as follows: the data format is unified to GeoJSON spatial data format, and Beijing time is used as the unified spatiotemporal reference. The timestamps of all data are aligned and the spatial coordinates are unified (WGS84 coordinate system). The 3σ criterion is used to remove outliers such as location point jumps and speed exceeding limits in the trajectory data. Linear interpolation is used to complete the short-term missing data within 10 seconds. Finally, an initial traffic data asset library with unified spatiotemporal reference and standardized format is formed.

[0020] Step 2: Based on Kalman filtering and combined with road network topology constraints, perform noise reduction and positioning calibration on multi-source trajectory data. Constructing a Discrete Linear Kalman Filter Model: State Variables Where x and y are the two-dimensional plane coordinates of the vehicle, v is the vehicle's speed, and θ is the vehicle's heading angle; the state equation is: Wherein, the sampling period T=1s, and the state transition matrix A is set based on the uniform vehicle motion model, specifically as follows: The control input matrix B is a 4th-order identity matrix. =0, The noise is Gaussian white noise with a mean of 0; the observation equation is: Wherein, the observation matrix H is a 4th-order identity matrix. For observation and positioning data from various data sources, This is Gaussian observation noise with a mean of 0.

[0021] Introducing road network topology constraints: Based on the preprocessed road network data, a Hidden Markov Model is used to complete the initial matching of trajectory points and road network segments. The observation probability is calculated based on the vertical distance from the trajectory point to the road segment, and the transition probability is calculated based on the connectivity of adjacent road segments and vehicle speed. The preset distance threshold is 50m. Abnormal positioning points that deviate from the road network by more than 50m are removed. The state prediction value of the Kalman filter is corrected based on the direction and connectivity of the road segment to ensure that the predicted trajectory conforms to the actual traffic rules of the road network.

[0022] Iterative denoising through filtering: The multi-source trajectory data is denoised and calibrated point-by-point using a Kalman filter prediction-update iterative process. This eliminates positioning noise caused by GPS signal obstruction and sampling bias, outputting a calibrated, standardized trajectory dataset. Testing showed that the calibrated trajectory positioning error in this embodiment was reduced from the original 3.2m to 1.5m.

[0023] Step 3: Based on the space-time dual-branch attention mechanism, complete the dynamic weight adaptive allocation and fusion processing of multi-source data. A dual-branch attention mechanism model is constructed, which includes spatial attention and temporal attention branches. The input is calibrated multi-source trajectory data, and the output is the dynamic fusion weights of three data sources: floating car, ETC, and checkpoint. After the weights are calculated, they are normalized by the Softmax function to ensure that the sum of the fusion weights of each data source is 1.

[0024] Weight allocation logic: Spatial Attention Branch: Calculates the congestion level of a road segment (quantitative index) based on the average vehicle speed and traffic saturation of the road segment. The higher the congestion level, the higher the criticality of the road network, and the higher the weight of floating car trajectory data with high sampling frequency in the corresponding spatial area. Time-attention branch: Based on the division of peak / off-peak periods and combined with the traffic flow change pattern, the traffic flow fluctuates greatly during peak periods, so the higher the weight of floating car trajectory data with high time resolution, the higher the weight of data from fixed locations such as ETC and checkpoints during off-peak periods when traffic flow is stable.

[0025] Actual weight allocation: For congestion-sensitive urban arterial roads and expressways, the weight of floating vehicle trajectory data is adaptively increased to 0.7; for unobstructed highway sections, the weight of ETC gantry data is adaptively increased to 0.65. For the morning peak hours of 7:00-9:00 and the evening peak hours of 17:00-19:00, the weight of floating vehicle trajectory data is increased to 0.68; for off-peak hours, the weight of ETC and checkpoint data is increased to 0.6.

[0026] Endpoint value test: To verify the rationality of the weight range, four endpoint value test scenarios were set up, and the results all met the accuracy requirement of travel time calculation error ≤ 5%. Scenario 1: Congestion-sensitive area, floating car trajectory data weight 0.6, travel time calculation error 4.7%; Scenario 2: In congestion-sensitive areas, the floating car trajectory data weight is 0.8, and the travel time calculation error is 4.1%. Scenario 3: During off-peak hours, the weight of ETC gantry data is 0.5, and the travel time calculation error is 4.5%. Scenario 4: During off-peak hours, the weight of ETC gantry data is 0.7, and the travel time calculation error is 4.3%.

[0027] The multi-source trajectory data is weighted and fused using dynamically assigned weights, outputting a fused full vehicle trajectory dataset. Comparative testing shows that the fusion method in this embodiment reduces the travel time calculation error from 12.3% to 4.2% compared to the traditional fixed-weight fusion method, significantly improving fusion accuracy.

[0028] Step 4: Integrate spatial matching and global correlation of data to build standardized smart transportation data assets. ArcGIS Pro was used to perform spatial matching between the fused trajectory data and the road network. Based on the road network topology data, the trajectory points were accurately mapped to the corresponding road segments and lanes. According to the test, the spatial matching accuracy of this embodiment reached 98.7%.

[0029] Based on a unified spatiotemporal benchmark, and using road segment numbers and timestamps as association keys, the system achieves full-domain association of merged vehicle trajectory data, road segment traffic operation status data, and meteorological and traffic event data. It establishes a full-dimensional mapping relationship of "single vehicle trajectory - traffic status of the road segment - corresponding environmental influencing factors," ultimately forming a structured, reusable, and traceable standardized smart transportation data asset, which is stored in a traffic data asset repository to support subsequent traffic analysis and business application calls.

[0030] Step 5: Integrate end-to-end computing power adaptation and parallel acceleration optimization Adaptation and optimization are carried out in three aspects: hardware, software, and algorithms, to ensure the real-time management and control requirements of intelligent transportation. Hardware computing power adaptation: The core server adopts an Intel Xeon Gold 6430 processor, configured with 256GB DDR4 memory and 8TB NVMe SSD hard drive to meet the parallel processing needs of millions of trajectory points; the edge uses industrial-grade embedded ARM computing devices, deployed at intersections and gantries, to adapt to complex outdoor operating environments.

[0031] Software framework adaptation: Basic data preprocessing is completed using Python's Pandas and NumPy libraries, spatial matching is performed using ArcGISPro, and the algorithm architecture of the fusion model is built based on TensorFlow 2.10.

[0032] GPU parallel acceleration optimization: Using NVIDIA A100 graphics cards, the core algorithms of Kalman filter iteration and attention mechanism inference are optimized for parallel acceleration based on the CUDA12.0 architecture.

[0033] Performance testing showed that the optimized data fusion processing efficiency increased from 200,000 records / hour to 1,020,000 records / hour, and the inference latency for single-batch data fusion remained stable within 180ms, fully meeting the real-time processing requirements of intelligent transportation.

[0034] Example 2: Multi-source data fusion and data asset construction in small-scale scenarios on highways Using a section of an intercity expressway as the application scenario, 50,000 floating car trajectory data, 10 ETC gantry vehicle passage data, 8 road checkpoint capture data, as well as road network topology data and data from 3 meteorological monitoring stations along the route were collected.

[0035] The method described in this invention is used for processing. Step 1 preprocessing completes the unification of spatiotemporal references. Step 2 noise reduction and calibration reduce the trajectory positioning error to 1.4m. Step 3 dynamic weighting and fusion reduces the travel time calculation error to 3.8%. Step 4 uses ArcGIS Pro to achieve spatial matching (accuracy 98.2%) and complete the association between "vehicle-road-environment". Step 5 acceleration optimization is completed based on lightweight hardware configuration (core server Intel Xeon Gold 6430 + 128GB memory). The single-batch inference latency is 150ms, and the data processing efficiency is 850,000 records / hour.

[0036] This embodiment verifies the applicability of the method of the present invention in small data volume and single scenario, and has good scenario generalization ability, which can adapt to the data fusion needs of smart transportation road networks of different scales.

[0037] Comparison Experiment of Existing Technologies To further verify the technical effectiveness of this invention, three sets of control experiments were set up, employing a traditional fixed-weight fusion scheme, a pure deep learning fusion scheme, and the scheme of this invention, respectively. Tests were conducted on the same city-level intelligent transportation dataset and under the same hardware environment. The test results are shown in the table below: Experimental results show that the present invention achieves higher fusion accuracy, lower inference latency and higher processing efficiency at the same time, which solves the technical contradiction that the existing technology cannot balance "accuracy priority" and "efficiency priority", and has the best scenario adaptability.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing multi-source data assets for intelligent transportation, characterized in that, Each step is executed based on a unified spatiotemporal reference and follows a progressive processing logic of noise reduction followed by weighting, specifically including the following steps: S1. Collect multi-source heterogeneous data on intelligent transportation and preprocess it to build an initial transportation data asset library; S2. Based on Kalman filtering and combined with road network topology constraints, noise reduction and positioning calibration are performed on multi-source trajectory data to eliminate positioning deviations caused by data noise and environmental interference. S3. Based on the space-time dual-branch attention mechanism, the fusion weights of the denoised and calibrated multi-source data are dynamically and adaptively allocated to complete the high-precision fusion processing of multi-source data. S4. Perform road network spatial matching on the fused full data to complete the multi-dimensional spatiotemporal association of "vehicle-road-environment" and build standardized smart transportation data assets.

2. The method according to claim 1, characterized in that, The multi-source heterogeneous data collected in S1 includes full heterogeneous data in vehicle operation dimension, road infrastructure dimension, and traffic environment dimension; the vehicle operation dimension data includes floating car trajectory data, ETC gantry vehicle passage data, and checkpoint capture data; the road infrastructure dimension data includes road network topology data and road segment attribute data; and the traffic environment dimension data includes meteorological data, construction road occupation data, and traffic incident data; the preprocessing includes data format unification, spatiotemporal benchmark alignment, outlier removal, and missing value completion.

3. The method according to claim 1, characterized in that, S2 specifically includes the following sub-steps: S21. Construct the state equation and observation equation of the discrete linear Kalman filter model, with the two-dimensional position, speed and heading angle of the vehicle as state variables, and the trajectory positioning data collected from each data source as observation variables. S22. Introduce road network topology constraints, use a hidden Markov model to match the original trajectory points to the corresponding road network segments, remove abnormal positioning points that deviate from the preset distance threshold of the road network, and correct the state prediction value of the Kalman filter based on the road segment orientation and connectivity of the road network. S23. Through the prediction and update iteration process of Kalman filtering, point-by-point noise reduction processing of multi-source trajectory data is completed, and the calibrated standardized trajectory dataset is output.

4. The method according to claim 1, characterized in that, The spatial-temporal dual-branch attention mechanism in S3 includes a spatial attention branch and a temporal attention branch. The spatial attention branch adaptively allocates the fusion weights of each data source in different spatial regions based on quantitative indicators of road segment congestion level and road network criticality. The time attention branch adaptively allocates fusion weights for each data source within different time dimensions based on peak / off-peak period division and traffic flow change patterns.

5. The method according to claim 4, characterized in that, In step S3, the calibrated multi-source trajectory data is used as the input of the dual-branch attention mechanism, and the dynamic fusion weights of each data source are used as the output. After normalization, the sum of the fusion weights of each data source is 1. Based on the output weights, the weighted fusion of multi-source data is completed to obtain the fused full vehicle trajectory dataset.

6. The method according to claim 1, characterized in that, In S4, geographic information system tools are used to complete the spatial matching of the fused trajectory data with the road network; based on a unified spatiotemporal benchmark, using road segment number and timestamp as the association key, the entire domain association of vehicle driving trajectory, road segment traffic status, and environmental influencing factors is completed, forming a structured, reusable, and traceable standardized smart transportation data asset.

7. The method according to claim 6, characterized in that, The geographic information system tool mentioned is ArcGIS Pro, and the spatial matching accuracy between trajectory data and road network achieved by this tool is no less than 98%.

8. The method according to claim 1, characterized in that, It also includes step S5, optimizing the entire data fusion process by adapting computing power and accelerating parallel processing, specifically including: S51, Hardware computing power adaptation: The core server uses a multi-core high-performance server processor and a large memory configuration to meet the parallel processing needs of millions of trajectory points. It is equipped with a high-speed SSD hard drive to ensure high-frequency data reading and writing. The edge end uses industrial-grade embedded computing devices to adapt to complex outdoor operating environments. S52. Software framework adaptation: Basic data processing is completed using Python's Pandas and NumPy libraries, and the algorithm architecture of the fusion model is built based on the deep learning framework. S53, GPU parallel acceleration optimization: NVIDIA high-performance computing graphics cards are used to perform CUDA parallel acceleration optimization on the fusion core algorithm.

9. The method according to claim 8, characterized in that, The core server in S51 is configured with no less than 256GB of memory and 8TB of high-speed SSD; the deep learning framework in S52 is the TensorFlow framework.

10. The method according to claim 9, characterized in that, The core server in S51 uses an Intel Xeon Gold 6430 processor; the S53 uses an NVIDIA A100 graphics card to perform parallel acceleration optimization of the fusion core algorithm.

Images