Video image recognition and charging data fused traffic state monitoring method and system

Abstract

The invention discloses a traffic state monitoring method and system fusing video image recognition and charging data, and the method comprises the steps: obtaining multi-source traffic data, carrying out the preprocessing of the multi-source traffic data, and generating preprocessing data; identifying the pre-processed data to generate vehicle related information; fusing and matching the preprocessed data and the vehicle related information to generate fused traffic data; analyzing the fused traffic data to generate traffic state information; a traffic management and decision support result is generated, and real-time congestion early warning and traffic accident early warning are provided; and mining and analyzing the historical traffic data to generate a traffic flow rule and accident rule analysis result. According to the invention, comprehensive, accurate and real-time monitoring and analysis of the traffic state are realized, powerful support is provided for traffic management and decision making, and the problems of difficult integration of multi-source data, insufficient real-time performance and coverage of state monitoring and imbalance of decision support and cost efficiency are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic state monitoring, and particularly to a traffic state monitoring method and system that integrates video image recognition and toll data. Background Art

[0002] Currently, the traffic state monitoring on highways mainly adopts a technical means that combines fixed-point video monitoring, radar detection, and floating car data. Specifically, fixed-point video monitoring can capture road images in real time, providing intuitive visual information for traffic state analysis; radar detection uses its high-precision speed measurement and ranging capabilities to accurately monitor the driving speed and spacing of vehicles; floating car data reflects the dynamic changes of traffic flow by collecting information of vehicles in motion. Through object detection algorithms, vehicle targets can be accurately identified from video monitoring and radar detection data, and key information such as the position and speed of vehicles can be obtained; map matching technology combines the driving trajectories of vehicles with electronic maps to achieve accurate positioning of vehicle positions and path restoration; statistical analysis methods process various types of data collected, extract traffic flow parameters such as flow, density, speed, etc., and evaluate the traffic state based on this.

[0003] However, the existing technical system has significant defects: (1) Difficulty in integrating multi-source heterogeneous data: Video, radar, and toll system data have low fusion accuracy due to inconsistent spatio-temporal benchmarks (timestamp deviation ≥ 3 seconds, coordinate error > 10 meters) and format differences, and the misjudgment rate of cross-system trajectory matching exceeds 30%; (2) Insufficient real-time performance and coverage of state monitoring: The deployment interval of fixed monitoring devices is large (≥ 2 km), short-term traffic flow mutations cannot be captured, the penetration rate of floating car data is low (< 20%) forming monitoring blind spots, and the missed judgment rate of hidden congestion reaches 45%; (3) Imbalance between decision support and cost efficiency: The analysis of historical data is limited to statistical reports, lacking causal association mining and predictive strategies, and the single-point deployment cost of radar / geomagnetic devices exceeds 50,000 yuan, and the independent operation of multiple systems leads to a 60% increase in integration and development costs. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a traffic state monitoring method and system that integrates video image recognition and toll data to solve the problems of difficult integration of multi-source data, insufficient real-time performance and coverage of state monitoring, and imbalance between decision support and cost efficiency existing in the existing traffic state monitoring technology.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a traffic status monitoring method integrating video image recognition and toll data, comprising: Collect ETC / MTC vehicle passing records at toll booths, video surveillance images of key road power connections, and floating vehicle data to obtain multi-source traffic data. Perform preprocessing on the multi-source traffic data, including image enhancement, denoising, format unification, and data cleaning, to generate preprocessed data. Vehicle recognition technology based on the fusion of deep learning and optical character recognition is used to identify pre-processed data and generate vehicle-related information; The pre-processed data and vehicle-related information are fused and matched, that is, the pre-processed data is time-space aligned to unify the time and space dimensions of data from different sources, and the standardized video trajectory data information is matched with the ETC / MTC vehicle passing records and floating vehicle trajectories based on the vehicle trajectory matching model of the graph neural network to generate integrated vehicle trajectory information. Through data quality assessment and abnormality filtering, the vehicle trajectory information is eliminated of abnormal data to generate fused traffic data; Analyze the integrated traffic data to generate traffic status information; Apply traffic status information through visual display to generate traffic management and decision support results, and provide real-time congestion warning and traffic accident warning; Mining and analyzing historical traffic data can generate analysis results of traffic flow patterns and accident patterns.

[0007] As a preferred solution of the traffic status monitoring method based on video image recognition and toll data fusion of the present invention, obtaining multi-source traffic data includes: Collect ETC / MTC vehicle passing records at toll stations to obtain the time, license plate number and vehicle model information of vehicles passing through toll stations; Collect video surveillance images of key nodes to obtain the driving conditions of vehicles on the road; Collect floating vehicle data to obtain vehicle speed and trajectory information.

[0008] As a preferred solution of the traffic status monitoring method based on video image recognition and toll data fusion of the present invention, the pre-processed data is identified and the vehicle related information is generated, which includes: Perform vehicle detection on preprocessed data based on a deep learning model and output vehicle bounding box coordinates and confidence information; The vehicle detection frames in consecutive frames are associated through a multi-target tracking algorithm to generate initial trajectory fragments, and the broken trajectories are repaired by smooth interpolation to form a complete vehicle spatiotemporal trajectory sequence information; Convert pixel coordinates to geographic coordinates according to the camera calibration parameters, and generate standardized video trajectory data information including timestamp, longitude and latitude, speed, and motion state; The generated preprocessed data includes: Perform image enhancement and denoising operations on the video surveillance images to generate high-quality image data; Unify the formats and clean the ETC / MTC vehicle passing records and floating vehicle data to generate data with consistent formats and no redundancy.

[0009] As a preferred solution of the traffic state monitoring method based on video image recognition and toll data fusion of the present invention, wherein: the fusion and matching of the preprocessed data and vehicle-related information includes: Perform spatio-temporal alignment on the preprocessed data to unify the time and space dimensions of data from different sources; Based on the vehicle trajectory matching model of the graph neural network, match the standardized video trajectory data information with the ETC / MTC vehicle passing records and floating vehicle trajectories to generate integrated vehicle trajectory information; Eliminate abnormal data from the vehicle trajectory information through data quality assessment and anomaly filtering to generate fused traffic data.

[0010] As a preferred solution of the traffic state monitoring method based on video image recognition and toll data fusion of the present invention, wherein: the analysis of the fused traffic data to generate traffic state information includes: Based on the road section traffic flow density calculation model, obtain the traffic load information of the road section; Adopt a congestion state grading recognition algorithm to generate congestion degree grading information of the road section; Predict the spatio-temporal evolution trend of the traffic state to generate prediction information of the traffic state.

[0011] As a preferred solution of the traffic state monitoring method based on video image recognition and toll data fusion of the present invention, wherein: generate traffic management and decision support results, and provide real-time congestion warnings and traffic accident warnings, including: Display the traffic situation through multi-dimensional visualization, including: maps, charts, and real-time video forms; Provide real-time congestion warnings and traffic accident warnings to support traffic management decisions.

[0012] As a preferred solution of the traffic state monitoring method based on video image recognition and toll data fusion of the present invention, wherein: the mining and analysis of historical traffic data to generate traffic flow pattern and accident pattern analysis results includes: Grid and aggregate the historical traffic data according to the unified spatio-temporal dimensions, extract traffic flow, speed, and driving behavior characteristics and label accident tags to generate a multi-dimensional analysis data set with spatio-temporal coordinates; Based on a multi-dimensional analysis dataset, analyze the regular pattern of periodic vehicle flow changes. At the same time, quantify the associated influence weights of external factors such as weather and events on flow fluctuations, and construct a traffic flow prediction neural network model that can predict the flow distribution in future time periods. Analyze the regular pattern of flow changes and establish an accident risk classification and early warning model that integrates real-time traffic flow characteristics. Spatiotemporally superimpose the traffic flow prediction neural network model and the accident risk early warning model to generate a visual decision-making map, and dynamically optimize the signal light control strategy, emergency resource deployment plan, and speed limit control rules based on the results of the regular pattern analysis to generate the analysis results of traffic flow patterns and accident patterns.

[0013] In a second aspect, the present invention provides a traffic state monitoring system that integrates video image recognition and toll collection data, including: a multi-source data acquisition module for collecting ETC / MTC data, video image data, and floating vehicle data. A preprocessing module configured with a GPU acceleration unit to achieve data cleaning and compression. A data fusion and matching module deployed with a graph neural network computing unit to achieve real-time trajectory matching. A traffic state inference module integrated with a machine learning model library to support state prediction. An application service layer that provides a visual human-computer interaction interface and an early warning interface.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: By unifying the spatio-temporal reference and format, the present invention effectively solves the problem of low fusion accuracy, and significantly reduces the misjudgment rate of cross-system trajectory matching; Through the fusion technology of deep learning and optical character recognition, the accuracy of vehicle recognition and the integrity of trajectory generation are improved, providing a more reliable data basis for subsequent analysis; Through the spatio-temporal alignment and graph neural network trajectory matching technology, precise fusion of multi-source data is achieved, effectively filling the monitoring blind spots caused by the large interval of fixed monitoring devices and the low penetration rate of floating vehicle data, and significantly reducing the missed judgment rate of hidden congestion; Through traffic flow density calculation, congestion level identification and spatio-temporal evolution prediction, comprehensive traffic state information is generated, providing more accurate decision-making support for traffic management and making up for the deficiencies in historical data analysis in the prior art; Through multi-dimensional visual display and real-time warning functions, the decision-making efficiency and the real-time nature of traffic management are improved; By visualizing the traffic situation in multiple dimensions, the efficiency of traffic management and the scientific nature of decision-making are improved; By providing real-time congestion warnings and traffic accident warnings, the impact of traffic congestion and accidents is reduced, ensuring road traffic safety and smoothness; By constructing a traffic flow prediction neural network model that can predict the future traffic flow distribution, the laws of traffic flow and accidents can be deeply explored, providing a scientific basis for traffic planning and management; By analyzing the traffic flow change law, the whole process management from traffic flow monitoring to accident risk warning and then to optimized decision-making is realized, improving the safety and operation efficiency of the traffic system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 It is a schematic diagram of the overall process of a traffic state monitoring method based on video image recognition and toll data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] Embodiment 1 Refer to Figure 1, which is an embodiment of the present invention, provides a traffic status monitoring method for video image recognition and toll data fusion, including: S100: Collect data on ETC / MTC vehicle passing records at toll stations, video surveillance images of key roads powered on, and probe vehicle data to obtain multi-source traffic data, and perform preprocessing on the multi-source traffic data, including image enhancement, denoising operations, format unification, and data cleaning, to generate preprocessed data; S200: Based on a vehicle recognition technology that combines deep learning and optical character recognition, identify the preprocessed data to generate vehicle-related information; S300: Integrate and match the preprocessed data and vehicle-related information, that is, perform spatio-temporal alignment on the preprocessed data to unify the time and space dimensions of data from different sources, and based on a vehicle trajectory matching model of a graph neural network, match the standardized video trajectory data information with ETC / MTC vehicle passing records and probe vehicle trajectories to generate integrated vehicle trajectory information. By data quality assessment and anomaly filtering, abnormal data is removed from the vehicle trajectory information to generate fused traffic data; S400: Analyze the fused traffic data to generate traffic status information; S500: Apply the traffic status information through visual display to generate traffic management and decision support results, and provide real-time congestion warnings and traffic accident warnings; S600: Mine and analyze historical traffic data to generate analysis results on traffic flow patterns and accident patterns.

[0019] It should be noted that the current traffic status monitoring faces the following core challenges: First, the problem of multi-source data fusion includes, For the vehicle passing records of ETC / MTC (Electronic Toll Collection System; Manual Toll Collection System), the time accuracy error is about 1 second. There are spatio-temporal benchmark differences between video surveillance images and probe vehicle GPS data with a sampling interval of 5 - 60 seconds, where the time deviation > 3 seconds and the coordinate error > 10 meters. These errors result in a cross-source vehicle trajectory matching success rate of less than 65%; Video images are also affected by light changes and occlusion. Among them, the undetected rate of vehicle detection at night > 35%, and the license plate recognition accuracy < 80%; Secondly, the real-time performance of status evaluation is insufficient, specifically including: Traditional methods rely on a single data source (such as only using video traffic statistics), and the monitoring delay for sudden changes in short-term traffic flow (such as congestion spread within 5 minutes caused by an accident) > 8 minutes; The penetration rate of floating car data is low (<20%), making it difficult to capture the traffic status of the entire road network in real time; Finally, the lack of intelligent decision-making support specifically includes: The historical data analysis is limited to statistical reports and cannot mine deep association rules such as "rainy / snowy weather + evening rush hour → the ramp accident rate increases by 72%"; The false alarm rate of the existing warning system is >25%, and it lacks the ability to predict the congestion propagation path.

[0020] Therefore, the present invention constructs a complete technical solution for traffic status monitoring based on the fusion of video image recognition and toll data through steps S100 - S600. By unifying the spatio-temporal reference and format, it effectively solves the problem of low fusion accuracy, and significantly reduces the misjudgment rate of cross-system trajectory matching; through the fusion technology of deep learning and optical character recognition, it improves the accuracy of vehicle recognition and the integrity of trajectory generation, providing a more reliable data basis for subsequent analysis; through spatio-temporal alignment and graph neural network trajectory matching technology, it realizes the precise fusion of multi-source data, effectively fills the monitoring blind spots caused by the large interval of fixed monitoring devices and the low penetration rate of floating car data, and significantly reduces the hidden congestion misjudgment rate; through traffic flow density calculation, congestion grading recognition, and spatio-temporal evolution prediction, it generates comprehensive traffic status information, providing more accurate decision-making support for traffic management, and making up for the deficiencies in historical data analysis in the prior art; through multi-dimensional visualization display and real-time warning functions, it improves the decision-making efficiency and the real-time nature of traffic management; through multi-dimensional visualization display of traffic situation, it improves the efficiency of traffic management and the scientific nature of decision-making; through providing real-time congestion warnings and traffic accident warnings, it reduces the impact of traffic congestion and accidents, ensuring road traffic safety and smoothness; through constructing a traffic flow prediction neural network model that can predict the future period traffic flow distribution, it can deeply explore the laws of traffic flow and accidents, providing a scientific basis for traffic planning and management; through analyzing the traffic flow change rules, it realizes the full-process management from traffic flow monitoring to accident risk warning and then to optimized decision-making, improving the safety and operation efficiency of the traffic system.

[0021] Embodiment 2 Refer to Figure 1 , which is an embodiment of the present invention. Based on the above embodiment, a traffic status monitoring method based on the fusion of video image recognition and toll data is provided.

[0022] In the embodiment of the present application, in step S100, the passing records of ETC / MTC at toll stations, the video surveillance images of key roads powered on, and the floating car data are collected to obtain multi-source traffic data, and the multi-source traffic data is preprocessed including image enhancement, denoising operation, format unification, and data cleaning to generate preprocessed data, including the following steps A1 - A2: A1: Obtaining multi-source traffic data includes: Collect the ETC / MTC vehicle passing records at toll stations to obtain the time, license plate number, and vehicle type information when the vehicle passes through the toll station; Collect video surveillance images at key nodes to obtain the driving conditions of vehicles on the road; Collect floating car data to obtain the driving speed and driving trajectory information of vehicles.

[0023] It should be noted that the collection of ETC / MTC vehicle passing records specifically includes obtaining vehicle passing data through the ETC / MTC system at toll stations, including accurate timestamps (±<0.1 second), license plate numbers (regular expression verification format such as "Jing A·12345"), and vehicle type classifications (small vehicles / large vehicles). The data storage format is structured CSV, and the processing volume per second is ≥1000 records.

[0024] The collection of video surveillance images includes deploying 4K resolution cameras at key nodes to obtain real-time vehicle driving images, with a coverage range of ≥500 meters per camera. Among them, image transmission uses H.265 encoding, reducing bandwidth occupancy by 50%.

[0025] The collection of floating car data includes collecting vehicle trajectory data through in-vehicle GPS / Beidou terminals, with a sampling frequency of 1Hz and a positioning accuracy of ±1.5 meters. Among them, the vehicle trajectory data includes vehicle speed, longitude and latitude, and timestamps, and the collected vehicle trajectory data is uploaded to the cloud database in real time.

[0026] A2: Generating preprocessed data includes, Performing image enhancement and denoising operations on video surveillance images to generate high-quality image data; Unifying the formats and cleaning the data of ETC / MTC vehicle passing records and floating car data to generate data with consistent formats and no redundancy.

[0027] It should be noted that the video surveillance image enhancement uses the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm to process low-light images, improving the night vehicle detection accuracy from 75% to 92%. Its mathematical expression is: ; Among them, is the output image; is the pixel value of the original input image at the coordinate (x,y), with a value range of [0,255] (8-bit grayscale image); 32×32 means dividing the image into 32×32 pixel local blocks, and each block performs histogram equalization independently; To limit the histogram distribution of a single gray level and avoid noise amplification; The gray levels are divided into 256 discrete intervals, and the number of pixels is counted for each bin.

[0028] Data cleaning includes using regular expressions to verify the license plate number format for ETC / MTC data and achieving an error correction rate > 98%; Kalman filtering is used to smooth the floating car trajectories, abnormal points such as sharp accelerations are removed, and the abnormal trajectory filtering rate > 85% is achieved.

[0029] In the implementation mode of this application, in step S200, based on the vehicle recognition technology that combines deep learning and optical character recognition, the preprocessed data is recognized to generate vehicle-related information, including the following steps B1 - B3: B1: Vehicle detection is performed on the preprocessed data based on a deep learning model, and the vehicle bounding box coordinates and confidence information are output; Specifically, the deep learning model uses an improved YOLOv5s model (You Only Look Once version 5 small, the lightweight version of YOLOv5), and the CBAM attention module (Convolutional Block Attention Module) is introduced to improve the detection performance in occlusion scenarios. Among them, the model outputs the vehicle bounding box coordinates (x min , y min , x max , y max ) and the confidence (p ≥ 0.8).

[0030] Among them, the network architecture of YOLOv5s consists of three parts: Backbone (CSPDarknet53 (Cross Stage Partial Darknet53, cross-stage partial dark web 53)), Neck (PANet (Path Aggregation Network)), and Head (detection head): Backbone (main network): Extract multi-scale features through the cross-stage local network (CSP), and output feature maps of 3 levels (P3: 80×80, P4: 40×40, P5: 20×20).

[0031] Neck (neck network): Fuse shallow details and deep semantic information through the path aggregation network (PANet) to enhance the small target detection ability.

[0032] Head: Output the coordinates of the bounding box (xmin, ymin, xmax, ymax) and the confidence p, and perform non-maximum suppression (NMS) to filter redundant detection boxes.

[0033] Furthermore, the CBAM attention module is integrated by inserting CBAM (Convolutional Block Attention Module) after each C3 module in the Backbone, including channel attention, spatial attention, and feature reconstruction, as follows: Channel attention: Generate channel weight vectors through global average pooling and max pooling to enhance important channel features: ; Among them, represents the function of the channel attention mechanism, and the input is the feature map F; σ represents the sigmoid activation function, whose role is to compress the input value into the interval (0, 1) for generating weights; MLP (Multi-Layer Perceptron) is a simple feedforward neural network used to learn the correlation between channels; AvgPool (average pooling) performs global average pooling on the input feature map to obtain a channel descriptor for capturing the statistical information between channels; MaxPool (max pooling) performs global max pooling on the input feature map to obtain another channel descriptor for capturing the maximum activation information between channels.

[0034] Spatial attention: Generate a spatial weight matrix through channel compression and Sigmoid activation to focus on the target area; Feature reconstruction: Multiply the channel and spatial attention weights element-wise with the original feature map to obtain the reconstructed feature.

[0035] Furthermore, the optimization effect of the occlusion scenario includes: Occlusion sensitivity test: In the test set with an occlusion rate > 30% (such as a vehicle being partially occluded by a tree or other vehicles), the mAP@0.5 (mean Average Precision, the average precision mean at 0.5, that is, the overlapping area between the predicted box and the ground truth box needs to be ≥ 50% to be judged as a correct detection) value of the improved model is increased from 89.2% to 94.2%.

[0036] B2: Associate the vehicle detection boxes in consecutive frames through a multi-object tracking algorithm to generate initial trajectory segments, and perform smooth interpolation repair on the broken trajectories to form complete vehicle spatio-temporal trajectory sequence information; Specifically, an improved DeepSORT (Deep Learning + Simple Online and Realtime Tracking) algorithm framework is adopted. Robust tracking is achieved through the deep integration of ReID feature vectors and motion prediction, and an intelligent trajectory repair mechanism is equipped, specifically including: First, a 2D depth feature vector is extracted through a ReID network (Re-identification), and is collaboratively matched with the motion trajectory predicted by Kalman filtering. The matching content includes: Appearance feature: An improved ResNet50-IBN network (Residual Network 50 with Instance Batch Normalization) is used to extract discriminative vehicle features, and the attention to the license plate area is strengthened by 1.5 times feature weighting; Motion feature: Based on Kalman filtering of an 8D state vector (position + speed), the reasonable position range of the vehicle in the next frame is predicted; Joint matching: By weighted synthesis of appearance similarity (weight λ = 0.7) and motion consistency score, the optimal association is completed using the Hungarian algorithm to ensure that the ID switching rate is controlled below 12%.

[0037] Second, when a trajectory break (lost for 5 - 15 consecutive frames) is detected, the system starts a three-level repair mechanism: Kinematics verification: Check the speed continuity (Δv < 3m / s) and direction rationality (Δθ < 45°) of the trajectory before and after the break; Cubic spline interpolation: Construct a continuous spatial trajectory with time parameters, and force it to satisfy the endpoint motion constraints; Geofence verification: Ensure that the repaired trajectory conforms to the road geometry constraints and avoid conflicts with static obstacles.

[0038] Third, to ensure real-time performance, batch processing (batch = 16) of feature extraction is adopted to reduce the GPU memory occupancy by 37%; Develop a SIMD (Single Instruction Multiple Data) parallel implementation of Kalman filtering to achieve a processing speed of 38ms / frame; Dynamically adjust the matching threshold (λ ∈ [0.6, 0.8]) to automatically enhance the appearance feature weight when the light changes.

[0039] It should be noted that through the closed-loop processing flow of "detection - tracking - repair - verification", while maintaining the stability of the ID, the system achieves an IDSW < 12% (ID Switch Width, the proportion of the target identity switching times), effectively recovers the trajectory breakage caused by occlusion (success rate > 90%), and forms a complete and reliable vehicle behavior record. The performance comparison of its optimization results is shown in Table 1.

[0040]

[0041] B3: According to the camera calibration parameters, convert the pixel coordinates into geographical coordinates, and generate standardized video trajectory data information including timestamp, longitude and latitude, speed, and motion state. Specifically, camera calibration includes obtaining the internal parameter matrix K and the external parameter matrix R|t of the camera based on the Zhang Zhengyou calibration method, and establishing the mapping relationship from pixel coordinates (u, v) to geographical coordinates (X, Y). The formula is as follows: ; Among them, is the internal parameter matrix, which contains the internal parameters of the camera, such as focal length, principal point coordinates, etc., and describes the conversion relationship between pixel coordinates and the camera coordinate system. is the inverse matrix of the internal parameter matrix K. is the rotation matrix, which describes the rotation relationship of the camera coordinate system relative to the geographical coordinate system. is the inverse matrix of the rotation matrix R. is the translation vector, which describes the translation relationship of the camera coordinate system relative to the world coordinate system. (u, v) are the pixel coordinates in the image coordinate system. (X, Y) are the real-world coordinates in the geographical coordinate system. is the image coordinate represented in the homogeneous coordinate system.

[0042] Trajectory feature extraction includes generating standardized video trajectory data, including timestamp t, longitude and latitude (X, Y), instantaneous speed v (calculation method: ) and motion state labels (constant speed / lane change / hard braking).

[0043] In the embodiment of the present application, in step S300, the preprocessed data and vehicle-related information are fused and matched, that is, the preprocessed data is aligned in time and space to unify the time and space dimensions of data from different sources, and based on the vehicle trajectory matching model of the graph neural network, the standardized video trajectory data information is matched with the ETC / MTC vehicle passing records and floating vehicle trajectories to generate the integrated vehicle trajectory information. Through data quality assessment and anomaly filtering, abnormal data is removed from the vehicle trajectory information to generate fused traffic data, including the following steps C1 - C2: C1: Align the preprocessed data in time and space to unify the time and space dimensions of data from different sources; Specifically, the time and space alignment includes Timestamp calibration: Perform sliding window matching (window size = 1 second) on the ETC / MTC data, video trajectories, and floating vehicle trajectories to ensure that the time synchronization error < 200 ms; Spatial coordinate unification: Align the geographical coordinates of the video trajectories with the floating vehicle GPS coordinates to the same coordinate system, with a residual < 3 meters.

[0044] C2: Based on the vehicle trajectory matching model of the graph neural network, match the standardized video trajectory data information with the ETC / MTC vehicle passing records and floating vehicle trajectories to generate the integrated vehicle trajectory information; Specifically, the construction of the vehicle trajectory matching model based on the graph neural network includes: Adopt the graph attention network (GAT), where the nodes represent the trajectory point features (time, position, speed), and the edge weights are calculated as spatio-temporal similarity: ; Among them, , C3: Through data quality assessment and anomaly filtering, remove abnormal data from the vehicle trajectory information to generate fused traffic data.

[0045] Specifically, the anomaly filtering includes: Detect abnormal trajectories (such as sudden speed changes and path deviations) based on the LOF (Local Outlier Factor) algorithm, with a recall rate > 87%.

[0046] In the embodiment of the present application, in step S400, the fused traffic data is analyzed to generate traffic state information, including the following steps D1 - D2: D1: Based on the road section flow density calculation model, obtain the traffic load information of the road section; Specifically, the formula of the road section flow density calculation model is as follows: ; Among them, N is the number of vehicles, L is the road section length, CV is the speed variation coefficient, and α = 0.2 is the correction factor.

[0047] D2: Adopt a congestion state classification and recognition algorithm to generate congestion level classification information for road segments; Specifically, congestion classification and prediction include four levels of congestion determination: Divide the congestion level based on the CV threshold: Unobstructed (CV ≤ 0.1); Slow (0.1 < CV ≤ 0.3); Congested (0.3 < CV ≤ 0.5); Severely congested (CV > 0.5).

[0048] D3: Predict the spatio-temporal evolution trend of traffic states and generate prediction information on traffic states.

[0049] Among them, predicting the spatio-temporal evolution trend of traffic states includes using a spatio-temporal graph convolutional network (ST-GCN), inputting historical 15-minute data, outputting a congestion heat map for the next 30 minutes, with the mean absolute error (MAE) < 2 km / h.

[0050] In an alternative implementation, the method for generating traffic state information in step S400 can also be targeted at predicting the spatio-temporal evolution of traffic states. By using a spatio-temporal graph convolutional network (Spatio-Temporal Graph Convolutional Network, ST-GCN), more accurate congestion propagation prediction can be achieved by modeling the road network topology and time dependence. Specifically, it includes: Construct a graph to represent the traffic network, and define each road segment or intersection in the traffic network as a node. Define the edges in the graph according to the actual road connection relationships to represent the connections between road segments; Construct an adjacency matrix to represent the graph, where the elements in the matrix represent the connection strength or weight between nodes; Select data useful for traffic state prediction, such as historical traffic flow, speed, weather conditions, etc., and extract features from this data, including statistical features, time series features, etc.; Preprocess the data after feature extraction and divide the preprocessed data into a training set, a validation set, and a test set; Use the training set data to train the ST-GCN model, adjust the model parameters through backpropagation and optimization algorithms, and evaluate the model performance on the validation set. Adjust the model parameters and structure to optimize the performance; Input real-time or recent traffic data into the trained ST-GCN for traffic state prediction and output the prediction results.

[0051] Analyze the prediction results to obtain traffic state information, and use the traffic state information for traffic management decisions, such as adjusting signal timings, issuing traffic warnings, etc., and visually display the traffic state information and prediction results in the form of GIS (Geographic Information System) maps, charts, etc.

[0052] In the embodiment of the present application, in step S500, the traffic state information is applied through visual display to generate traffic management and decision support results, and real-time congestion warnings and traffic accident warnings are provided, including the following steps E1 - E2: E1: Visually display the traffic situation through multi-dimensional visualization, including in the forms of maps, charts, and real-time videos; E2: Provide real-time congestion warnings and traffic accident warnings to support traffic management decisions.

[0053] It should be noted that the multi-dimensional visualization display includes, Three-dimensional traffic situation map: Superimpose the real-time video stream and the congestion heat map, where red indicates the congested area (speed < 20 km / h) and green indicates the unobstructed area (speed > 60 km / h).

[0054] The decision support interface for supporting traffic management decisions includes: providing a RESTful API (Representational State Transfer Application Programming Interface), supporting dynamic regulation of traffic signals and ensuring a response time < 1 second, and reducing the false alarm rate of accident warnings to 12%.

[0055] In the embodiment of the present application, in step S600, the historical traffic data is mined and analyzed to generate analysis results of traffic flow patterns and accident patterns, including the following steps F1 - F4: F1: Aggregate the historical traffic data in a grid manner according to a unified spatio-temporal dimension, extract traffic flow, speed, and driving behavior characteristics and label accident tags to generate a multi-dimensional analysis data set with spatio-temporal coordinates; F2: Based on the multi-dimensional analysis data set, analyze the periodic variation law of vehicle traffic flow, and at the same time quantify the correlation influence weights of external factors such as weather and events on traffic flow fluctuations, and construct a traffic flow prediction neural network model that can predict the traffic flow distribution in future time periods; F3: Analyze the traffic flow change law and establish an accident risk grading and warning model that integrates real-time traffic flow characteristics; F4: Spatially and temporally superimpose the traffic flow prediction neural network model and the accident risk early warning model to generate a visual decision-making map, and dynamically optimize the signal light control strategy, emergency resource deployment plan, and speed limit control rules based on the results of pattern analysis to generate the analysis results of traffic flow patterns and accident patterns.

[0056] Among them, the in-depth mining of historical traffic data includes Association rule discovery, using the FP-Growth algorithm (Frequent Pattern Growth Algorithm): Mine rules such as "heavy rain weather + evening rush hour → ramp accident rate + 72%", with a support > 0.3 and a confidence > 0.8.

[0057] Traffic flow prediction model: Build a traffic flow prediction network based on LSTM (Long Short-Term Memory), input historical 72-hour data, and output the traffic flow distribution in the next 24 hours, with a root mean square error < 50 veh / h.

[0058] Dynamic policy optimization includes Reinforcement learning control, using the DQN algorithm (Deep Q-Network Algorithm) to optimize the signal light timing, and the traffic efficiency during peak hours is increased by 18%. The policy update period is 5 minutes, and the quality value function is expressed as: ; where s is the traffic state (traffic flow, speed, congestion level); a is the signal light action (cycle extension / shortening); Q(s,a): represents the Q value of executing action a in state s, that is, the expected return of this action in this state; E represents the expected value, which is used to calculate the average value of all possible results under a certain policy; represents the discount factor, γ is a value between 0 and 1, which is used to weigh the importance of future rewards; represents the immediate reward obtained at time step t, which is the feedback given by the environment after executing the action.

[0059] In summary, the present invention integrates the passing vehicle records of ETC / MTC at toll stations, video surveillance images, and floating vehicle data, achieving comprehensive collection and fusion of multi-source traffic data, and improving the accuracy and comprehensiveness of traffic state monitoring; adopts an improved YOLOv5s model and CBAM attention module, significantly enhancing the accuracy of vehicle detection; through an improved DeepSORT algorithm and intelligent trajectory repair mechanism, effectively restores trajectory breaks caused by occlusion, forming a complete and reliable vehicle behavior record; realizes timestamp calibration and spatial coordinate unification of data from different sources, ensuring the spatio-temporal consistency of data, and providing high-quality fused traffic data for subsequent analysis; based on the road section traffic flow density calculation model and congestion state classification and recognition algorithm, accurately obtains the traffic load information and congestion level classification information of the road section, providing a scientific basis for traffic management; uses a spatio-temporal graph convolutional network to predict the spatio-temporal evolution trend of traffic states, providing predictive support for traffic management; deeply mines and analyzes historical traffic data to generate analysis results of traffic flow patterns and accident patterns, providing a reference for traffic planning and management; uses reinforcement learning control and DQN algorithm to optimize signal timing, realizing dynamic optimization of traffic control strategies; through the above steps, the present invention realizes comprehensive, accurate, and real-time monitoring and analysis of traffic states, provides strong support for traffic management and decision-making, and solves the problems of difficult integration of multi-source data, insufficient real-time performance and coverage of state monitoring, and imbalance between decision-making support and cost efficiency.

[0060] Embodiment 3 The above is a schematic solution of the traffic state monitoring method for video image recognition and toll data fusion. It should be noted that the technical solution of the system for traffic state monitoring based on video image recognition and toll data fusion belongs to the same concept as the technical solution of the above traffic state monitoring method for video image recognition and toll data fusion. For the details not described in detail in the technical solution of the traffic state monitoring system based on video image recognition and toll data fusion in this embodiment, reference can be made to the description of the technical solution of the traffic state monitoring method for video image recognition and toll data fusion above.

[0061] This embodiment also provides a traffic state monitoring system for video image recognition and toll data fusion, including: A multi-source data collection module for collecting ETC / MTC data, video image data, and floating vehicle data; A preprocessing module configured with a GPU acceleration unit to implement data cleaning and compression; A data fusion and matching module deployed with a graph neural network computing unit to implement real-time trajectory matching; A traffic state inference module integrated with a machine learning model library to support state prediction; An application service layer providing a visual human-computer interaction interface and an early warning interface.

[0062] This embodiment also provides an electronic device, which is applicable to the situation of traffic state monitoring based on the fusion of video image recognition and toll data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the traffic state monitoring method based on the fusion of video image recognition and toll data as proposed in the above embodiment.

[0063] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the traffic state monitoring method based on the fusion of video image recognition and toll data as proposed in the above embodiment.

[0064] The storage medium proposed in this embodiment and the traffic state monitoring method based on the fusion of video image recognition and toll data proposed in the above embodiment belong to the same inventive concept. For the technical details not described in detail in this embodiment, reference can be made to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0065] From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software and necessary general-purpose hardware, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk, or an optical disc of a computer, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present invention.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A traffic status monitoring method integrating video image recognition and toll data, characterized in that: include: Collect ETC / MTC vehicle passing records at toll booths, video surveillance images of key road power connections, and floating vehicle data to obtain multi-source traffic data. Perform preprocessing on the multi-source traffic data, including image enhancement, denoising, format unification, and data cleaning, to generate preprocessed data. Vehicle recognition technology based on the fusion of deep learning and optical character recognition can identify pre-processed data and generate vehicle-related information; The pre-processed data and vehicle-related information are fused and matched, that is, the pre-processed data is time-space aligned to unify the time and space dimensions of data from different sources, and the standardized video trajectory data information is matched with the ETC / MTC vehicle passing records and floating vehicle trajectories based on the vehicle trajectory matching model of the graph neural network to generate integrated vehicle trajectory information. Through data quality assessment and abnormality filtering, the vehicle trajectory information is eliminated of abnormal data to generate fused traffic data; Analyze the integrated traffic data to generate traffic status information; Apply traffic status information through visual display to generate traffic management and decision support results, and provide real-time congestion warning and traffic accident warning; Mining and analyzing historical traffic data can generate analysis results of traffic flow patterns and accident patterns.

2. The traffic status monitoring method of video image recognition and toll data fusion as claimed in claim 1, characterized in that: The acquiring of multi-source traffic data comprises: Collect ETC / MTC vehicle passing records at toll stations to obtain the time, license plate number and vehicle model information of vehicles passing through toll stations; Collect video surveillance images of key nodes to obtain the driving conditions of vehicles on the road; Collect floating vehicle data to obtain vehicle speed and trajectory information; The generating preprocessing data comprises: Perform image enhancement and denoising operations on video surveillance images to generate high-quality image data; The format of ETC / MTC vehicle passing records and floating vehicle data is unified and data is cleaned to generate data with consistent format and no redundancy.

3. The traffic status monitoring method of video image recognition and toll data fusion as claimed in claim 2, characterized in that: The identifying of the pre-processed data and generating vehicle related information includes: Perform vehicle detection on preprocessed data based on a deep learning model and output vehicle bounding box coordinates and confidence information; The vehicle detection frames in consecutive frames are associated through a multi-target tracking algorithm to generate initial trajectory fragments, and the broken trajectories are repaired by smooth interpolation to form a complete vehicle spatiotemporal trajectory sequence information; According to the camera calibration parameters, the pixel coordinates are converted into geographic coordinates to generate standardized video trajectory data information including timestamp, longitude and latitude, speed and motion status.

4. The traffic status monitoring method of video image recognition and toll data fusion as claimed in claim 3 is characterized by: The analyzing of the fused traffic data to generate traffic status information includes: Based on the road section flow density calculation model, obtain the traffic load information of the road section; Adopt congestion status classification identification algorithm to generate congestion level classification information of road sections; Predict the spatiotemporal evolution trend of traffic status and generate traffic status prediction information.

5. The traffic status monitoring method of video image recognition and toll data fusion as claimed in claim 4, characterized in that: The generation of traffic management and decision support results and the provision of real-time congestion warning and traffic accident warning include: Display traffic situation through multi-dimensional visualization, including maps, charts and real-time video; Provide real-time congestion warning and traffic accident warning to support traffic management decision-making.

6. The traffic status monitoring method of video image recognition and toll data fusion as claimed in claim 5, characterized in that: The mining and analysis of historical traffic data to generate traffic flow and accident pattern analysis results includes: Grid-aggregate historical traffic data according to a unified spatiotemporal dimension, extract flow, speed, and driving behavior characteristics, and annotate accident labels to generate a multidimensional analysis data set with spatiotemporal coordinates; Based on the multidimensional analysis data set, the changing patterns of periodic vehicle traffic are analyzed, and the weights of the associated impacts of weather and external factors on traffic fluctuations are quantified to construct a traffic prediction neural network model that can predict traffic distribution in future time periods; Analyze the traffic flow variation pattern and establish an accident risk classification warning model integrating real-time traffic flow characteristics; The traffic prediction neural network model and the accident risk warning model are superimposed in time and space to generate a visual decision map, and the traffic light control strategy, emergency resource deployment plan and speed limit control rules are dynamically optimized based on the law analysis results to generate traffic flow law and accident law analysis results.

7. A traffic status monitoring system integrating video image recognition and toll data, applying the method according to any one of claims 1 to 6, characterized in that: include: Multi-source data acquisition module, used for collecting ETC / MTC data, video image data and floating vehicle data; Preprocessing module, equipped with GPU acceleration unit to achieve data cleaning and compression; Data fusion and matching module, deploys graph neural network computing units to achieve real-time trajectory matching; Traffic state inference module, integrating machine learning model library to support state prediction; The application service layer provides a visual human-computer interaction interface and an early warning interface.

Images