Traffic flow rapid sensing method based on multi-modal fusion

Abstract

The invention provides a traffic flow rapid sensing method based on multi-modal fusion. Comprising the following steps: S1, a multi-source heterogeneous sensor carries out collaborative data acquisition and space-time alignment; s2, edge side data preprocessing and feature level fusion; s3, multi-modal target perception and feature extraction based on an improved network model; s4, performing multi-modal decision-making level fusion and accurate state recognition; s5, performing global traffic state monitoring and event identification; and S6, performing model optimization and edge efficient deployment. According to the invention, through fusion of a heterogeneous sensor space-time cooperation mechanism, improved YOLOP network construction, a hierarchical decision fusion engine and an edge intelligent optimization technology, high-precision rapid perception and edge real-time processing of traffic flow under an all-weather condition are realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic perception and multimodal data fusion, specifically involving a method for rapid traffic flow perception based on multimodal fusion. Background Technology

[0002] With the rapid development and large-scale deployment of intelligent transportation systems, multi-sensor fusion perception technology, with its advantages of strong data complementarity, wide environmental adaptability, and high detection accuracy, has been widely applied in core scenarios such as traffic flow monitoring, target recognition and tracking, abnormal event early warning, and autonomous driving environmental perception. It has become a core technological support for intelligent traffic management, vehicle-road cooperation, and the safe operation of autonomous driving, gradually replacing traditional single-sensor perception solutions and driving the transportation system towards all-weather, high-precision, and intelligent upgrades. However, the environment of intelligent transportation scenarios is complex and ever-changing. Sensors need to operate under harsh conditions such as vehicle obstruction, sudden changes in lighting, severe weather (rain, snow, fog), and traffic congestion. Single sensors (vision, radar, etc.) are limited by their own physical characteristics and are prone to problems such as detection blind spots, data noise interference, and poor environmental adaptability, leading to irreversible errors in the perception system, such as missed detections, false detections, and positioning deviations. If the performance bottleneck of single sensors cannot be accurately overcome, it will not only reduce the management efficiency and decision-making reliability of intelligent transportation systems but also significantly increase safety risks such as collisions and traffic paralysis in scenarios such as autonomous driving and vehicle-road cooperation, posing a serious threat to traffic safety.

[0003] While research on multi-sensor fusion perception technologies has made some progress, their applicability in the complex real-world conditions of intelligent transportation remains significantly limited. The core problems of existing methods mainly focus on two aspects: First, most fusion algorithms rely on fixed feature fusion patterns and sensor calibration parameters. However, in intelligent transportation scenarios, sensors are susceptible to calibration drift due to vibration and temperature / humidity changes. Furthermore, different scenarios (urban roads, highways, tunnels) have varying requirements for fusion strategies, making it difficult for fixed patterns to adaptively adjust, leading to unstable fusion accuracy. Second, existing fusion technologies primarily focus on feature-level or decision-level fusion algorithm optimization, neglecting the computational resource constraints of edge devices. The inference speed of complex fusion models struggles to meet the demands of real-time intelligent transportation control and low-latency response for autonomous driving. Moreover, they lack sufficient differentiated perception capabilities for heterogeneous targets such as large vehicles, non-motorized vehicles, and pedestrians, easily resulting in insufficient feature extraction. Although deep learning technology has provided new directions for multimodal fusion in recent years, existing research has largely simplified the complexity of spatiotemporal sensor coordination, failing to fully consider the spatiotemporal asynchronicity and heterogeneity of multi-source data. This makes it impossible to fully capture the dynamic perception requirements in complex traffic scenarios, thus affecting the robustness and engineering applicability of the fusion system. Therefore, how to achieve spatiotemporal collaborative calibration and adaptive feature fusion of multiple sensors in complex traffic scenarios, while also considering the real-time requirements of edge devices, and constructing a high-precision, highly robust multi-sensor fusion perception system has become a critical issue that urgently needs to be addressed to ensure the safe and efficient operation of intelligent transportation systems.

[0004] Patent CN118333330A discloses a multimodal fusion perception method for urban traffic management. By collecting multi-source data from vision and radar, it constructs a feature-level fusion framework and extracts cross-modal correlation features to improve target detection accuracy. Simultaneously, it provides a standardized technical framework and implementation path for multimodal perception systems. This method focuses on multimodal data fusion optimization in urban traffic scenarios. Its core belongs to deep learning-based cross-modal feature fusion technology, exhibiting certain perception performance advantages in conventional traffic scenarios.

[0005] This invention proposes a "multi-sensor fusion perception method based on IMU tight coupling and an improved YOLOP network," addressing the multimodal perception needs in complex intelligent transportation scenarios. It focuses on three core issues: spatiotemporal collaborative calibration of multi-source sensors, adaptive feature extraction of heterogeneous targets, and real-time optimization of edge devices. The method constructs an IMU tight-coupled compensation calibration model, an improved YOLOP network architecture, and a hybrid precision optimization strategy. Compared to the aforementioned patented technologies, this invention emphasizes robustness improvement and engineering feasibility in complex scenarios. It solves the sensor calibration drift problem through IMU dynamic compensation, enhances the adaptability of heterogeneous target perception through network structure improvement, and overcomes the computational bottleneck of edge devices through hybrid precision optimization. This approach differs fundamentally in its technical approach, core objectives, and application scenarios. Experiments show that this invention significantly improves target recognition accuracy within a 300-meter detection range compared to existing methods, and the edge device inference speed meets the real-time management and control requirements of intelligent transportation. It provides more reliable and efficient perception technology support for intelligent transportation systems and can be widely adapted to various complex traffic scenarios such as urban roads, highways, and tunnels. Summary of the Invention

[0006] To address the aforementioned technical challenges, this invention proposes a rapid traffic flow perception method based on multimodal fusion. This method integrates precise time synchronization based on PTPv2, online Lie algebra calibration, multimodal feature extraction using an improved YOLOP network, and a hierarchical decision fusion engine to form a full-process intelligent processing architecture of "perception-fusion-decision". Furthermore, it constructs a real-time processing framework optimized by edge computing to achieve high-precision detection, tracking, and state recognition of traffic targets.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] This invention provides a method for rapid traffic flow perception based on multimodal fusion, comprising the following steps:

[0009] S1: Collaborative Data Acquisition and Spatiotemporal Alignment Using Multi-Source Heterogeneous Sensors:

[0010] Sensors are deployed using a layout of "radar centering and binocular visual angle". A nanosecond-level synchronization network based on PTPv2 protocol and GPS disciplined clock is constructed. Offline extrinsic parameter calibration is completed by minimizing reprojection error and SE(3) exponential mapping. In the online stage, the regenerating kernel Hilbert space (RKHS) similarity metric is used to achieve dynamic registration of visual features and radar signals. Combined with IMU pre-integration algorithm, pose error caused by road bumps is eliminated. The output is a spatiotemporally strictly aligned RGB image, semantic label and radar range-velocity spectrum data.

[0011] S2: Edge-side data preprocessing and feature-level fusion:

[0012] Data cleaning and standardization are performed in the edge computing unit, image noise is processed by median filtering, and distance correction is performed at the radar end. Primary trajectory prediction is performed based on Kalman filtering. The radar point cloud and visual detection box are initially matched through spatiotemporal correlation, and low-dimensional features containing target position and velocity are output, which effectively reduces data transmission bandwidth and improves system real-time performance.

[0013] S3: Multimodal target perception and feature extraction based on an improved network model:

[0014] An improved YOLOP network architecture is constructed: a C2F module and gradient splitting mechanism are introduced to enhance small target feature extraction, an SPPF module is used to implement adaptive multi-scale pooling, and ConvTranspose2D is used to optimize the upsampling process. A two-dimensional attention mechanism of coordinate attention (spatial) and ECA-Net (channel) is adopted to dynamically optimize feature weights. For the training process, Blender and GAN are used to generate extreme scene samples, and dynamic scale-aware adaptive focal loss (DSA-Focal Loss) is applied to improve detection robustness. At the same time, a dense correspondence between radar DBSCAN clustering and visual segmentation mask is established to achieve high-precision feature association.

[0015] S4: Multimodal decision-level fusion and accurate state recognition:

[0016] A multidimensional association cost function integrating IoU, optical flow velocity, and semantic similarity is constructed, and an adaptive weighted fusion strategy is implemented: visual weights are dynamically adjusted based on illumination and ambiguity, radar weights are calculated based on RCS (radar cross section) and the standard deviation of radial distance in point clouds, lane geometry constraint algorithm is introduced to correct visual perspective error, Doppler power spectrum main frequency symmetry analysis is used to optimize radar speed measurement, and finally the data quality health index (Q) is calculated. The fusion mode is dynamically switched or the online calibration process is triggered according to the score to ensure high confidence of the status output.

[0017] S5: Comprehensive Traffic Status Monitoring and Event Recognition

[0018] Based on the fusion results, full-segment monitoring is achieved: real-time statistics of traffic flow, speed, and trajectory; support for overspeed triggering warnings on illuminated vehicles and information boards; dedicated logic branches are designed for large vehicles to link active warnings, and events such as abnormal parking and low-speed driving are identified through reflection point density clustering and motion trajectory analysis, supporting traffic management and accident prevention;

[0019] S6: Model Optimization and Efficient Edge Deployment

[0020] During model training, class weights are dynamically adjusted, and targeted reinforcement is performed for large vehicles and difficult samples. During the deployment phase, the number of parameters is reduced through channel pruning. Combined with 8-bit quantization and TensorRT acceleration technology, low-latency real-time processing is achieved on edge computing platforms such as Jetson AGX Orin to meet the application needs of complex traffic scenarios.

[0021] As a preferred technical solution of the present invention, step S1 is specifically as follows:

[0022] Step 1: Multi-sensor hardware deployment and synchronous triggering, as detailed below:

[0023] Cameras, millimeter-wave radar, IMU and other sensors are deployed at traffic monitoring nodes to form a perception area that covers 300 meters horizontally and 8 lanes vertically. The monitoring nodes are located on roadside pillars and gantries. The camera types include visible light or infrared, and the millimeter-wave radar can be 77GHz FMCW.

[0024] A sandwich layout with "radar centered and dual cameras at a 15° angle" is adopted. Geometric constraints ensure that the coverage of the overlapping area of ​​the field of view is >85%. The optimal installation angle is calculated based on the field of view angle formula.

[0025] (1);

[0026] in, Lane width, To monitor distance, For safety margin, the binocular vision angle includes a 15° angle between the two cameras;

[0027] A synchronization network is constructed using the PTPv2 protocol, with a GPS-disciplined clock deployed as the master clock. Nanosecond-level synchronization is achieved through cable length compensation and spatial propagation delay compensation, as detailed in the following formula:

[0028] (2);

[0029] in, For sensor local timestamps, Where c is the length of the synchronization cable and c is the speed of light. ;

[0030] Step 2: Offline calibration and coordinate system one, as detailed below:

[0031] The radar-camera extrinsic parameter matrix is ​​obtained using the checkerboard calibration method, and the reprojection error is minimized.

[0032] (3);

[0033] in, For Lie algebra parameters, For the SE(3) exponent mapping, For Huber robust kernel function, The regularization coefficient is used.

[0034] With the camera's optical center as the origin of the world coordinate system, the radar coordinates (Xr, Zr) are directly mapped to the world coordinates (Xw, Zw). Yw is determined by measuring the radar's installation height using a laser rangefinder. The conversion formula is as follows:

[0035] (4);

[0036] Where h represents the radar installation height, addressing the issue of missing Y-axis data for the radar. ;

[0037] Step 3: Online dynamic registration and motion compensation, as detailed below:

[0038] Using fixed lane lines, road signs, and other features as baselines, a regenerative kernel Hilbert spatial similarity metric is constructed through visual recognition and radar reflection point matching.

[0039] (5);

[0040] in, For RBF kernel mapping functions, For feature space;

[0041] Using the acceleration / angular velocity data collected by the IMU, the pose changes of the sensor platform caused by wind vibration or vehicle bumps are calculated, and the IMU data is pre-integrated.

[0042] (6);

[0043] in, This is the measured value of angular velocity (rad / s). Zero bias for the gyroscope (calibrated using Allan variance);

[0044] Step 4: Output data with spatiotemporal alignment, as detailed below:

[0045] Align the data from each sensor with a unified timestamp and output a spatiotemporally synchronized multimodal data packet, including visual, radar, and auxiliary data;

[0046] The visual data includes RGB images (1920×1080@30fps) and lane line semantic labels, the radar data includes target range-velocity spectrum (0.1m / 0.1km / h resolution), and the auxiliary data includes IMU pose parameters, road surface temperature or humidity.

[0047] As a preferred technical solution of the present invention, step S3 is as follows:

[0048] The original YOLOP's BottleneckCSP module is replaced by the C2F module, which enhances the feature extraction capability for small targets and low-contrast traffic objects through cross-layer feature fusion and channel stitching.

[0049] The C2F module introduces a gradient splitting mechanism to restrict the direct transmission of shallow features, prevent information redundancy, and improve model training stability. The specific formula is as follows:

[0050] (7);

[0051] in, Represents low-level high-resolution features. Represents high-level semantic features. This represents depthwise separable convolution;

[0052] The SPPF module (Fast Spatial Pyramid Pooling) is used to replace the traditional SPP module, and serial max pooling (kernel=5×5, 9×9, 13×13) is used to achieve adaptive multi-scale feature fusion, reducing the amount of computation by 30% while expanding the receptive field.

[0053] The addition of Coordinate Attention encodes the feature map in both horizontal and vertical directions, enhancing its sensitivity to the aspect ratio of vehicles and improving the detection accuracy of large vehicles.

[0054] By replacing bilinear interpolation Upsample with ConvTranspose2D, the upsampling process is optimized through learnable parameters, avoiding the edge blurring problem of traditional interpolation methods and improving the continuity of lane lines in segmentation tasks.

[0055] Step 2: Two-dimensional attention mechanism and dynamic task allocation, as detailed below:

[0056] The coordinate attention module performs X / Y direction decomposition on the feature map to generate orientation-sensitive feature weights, as shown in the following formula:

[0057] Spacetime decomposition:

[0058] (8);

[0059] Spatiotemporal correlation modeling:

[0060] (9);

[0061] The intermediate attention response map after fusing spatiotemporal information;

[0062] Spatial weight generation:

[0063] (10);

[0064] in, Indicates the first Spatial location within each channel The relative importance weights of the corresponding features are normalized in the spatial dimension to enhance the response intensity of the vehicle body, edge contours, and interaction areas, while suppressing background interference information.

[0065] Efficient Channel Attention ECA-Net dynamically adjusts channel weights to highlight key features such as vehicle edges and textures while suppressing background noise. To avoid insufficient channel information caused by a single mean statistic, it introduces an efficient channel attention modeling method that integrates first-order and second-order statistics. The specific formula for channel statistics extraction is as follows:

[0066] (11);

[0067] in, This represents the global statistical descriptor for the c-th channel; The second term represents the mean of the c-th channel and characterizes the dispersion of channel activation. To adjust the contribution ratio of mean and variance information, the hyperparameter was optimized to 0.4.

[0068] Adaptive convolution kernel calculation:

[0069] (12);

[0070] A differentiable task weight controller is designed to dynamically adjust the loss function weights of the target detection, lane segmentation, and ranging subtasks based on real-time detection results. Focal Loss (α=0.8, γ=2) is used to optimize the detection head and enhance its learning of difficult samples such as occluded and small vehicles.

[0071] The proposed dynamic scale-aware adaptive focus loss DSA-Focal Loss optimization detection head is employed:

[0072] (13);

[0073] Here, α is 0.8, used to adjust the relative contribution of positive and negative samples to the total loss. To predict class probabilities, γ is set to 2, which is used to suppress the loss contribution of easily classified samples and enhance the model's robustness to targets in occluded, small-scale, and complex traffic scenarios. The value is 0.5, which is the scale enhancement coefficient used to adjust the degree of weight amplification for small-scale targets. This represents the effective receptive region area of ​​the target on the feature map, used to characterize the target's scale. , which is the scale normalization reference constant, is taken as the average value of all receptive regions of this category;

[0074] Step 3: Specific optimization strategies for large vehicles, as follows:

[0075] Blender was used to generate synthetic data for extreme scenarios such as container trucks in snowy weather and oil tankers in foggy weather. The GAN network generated partially occluded and deformed samples, and the proportion of large vehicle training samples was increased to 30%.

[0076] When the detection rate of large vehicles is below the threshold, their loss weight is automatically increased to resolve the class imbalance problem.

[0077] (14);

[0078] in, This represents the loss weight corresponding to the large vehicle in the t-th training period; This indicates the weight of large vehicle losses in the previous period. This is the time smoothing coefficient, used to control the degree of influence of historical weights on the current update, and is set to 0.1; This is the difficulty amplification factor, used to adjust the extent to which the detection difficulty increases the weight of the loss; it is set to 0.4. Let be the recall rate of large vehicles in the t-th training period. The average recall rate across all categories;

[0079] A length-to-width ratio sensitive anchor frame (8:1, 5:1) is added to the detection head to match special vehicle types such as trucks and trailers;

[0080] For large vehicles, a high-resolution feature map branch is used, and shallow detail features are fused using the C2F module to avoid the loss of small-scale features;

[0081] To address the feature misalignment caused by fixed-grid pooling, a sub-network is used to predict the sampling point offset. The specific formula is as follows:

[0082] (15);

[0083] in, This represents the coordinates of the original sampling points; FC represents a fully connected layer.

[0084] To address the inconsistency between radar and visual contours, a dense correspondence between radar point clouds and visual segmentation masks is established, with the specific formula as follows:

[0085] (16);

[0086] in, This represents the DBSCAN clustering results;

[0087] Step 4: Real-time feature output and intermediate results, as detailed below:

[0088] The detection targets include vehicle location (BBox), type, and confidence level, where vehicle type includes cars or trucks, etc.

[0089] Semantic segmentation is performed on lane lines and drivable area masks, where the drivable area mask resolution is 512×512.

[0090] The vertical distance is output based on inverse perspective transformation (IPM) of monocular vision, where the vertical distance error must be less than 5%@50m;

[0091] Output cross-layer feature fusion map and SPPF multi-scale feature map from the C2F module for model interpretability analysis.

[0092] As a preferred technical solution of the present invention, step S4 is specifically as follows:

[0093] Step 1: Cross-modal target association and confidence assessment, as detailed below:

[0094] Traditional methods only consider geometric location matching. To achieve multi-dimensional matching cost calculation, a cost function that integrates geometry, motion, and semantics is constructed. The specific formula is as follows:

[0095] (17);

[0096] in, Indicates the intersection and union of 3D bounding boxes. Indicates the visual optical flow estimation velocity. Indicates radar Doppler velocity, Indicates semantic similarity score;

[0097] Step 2: Adaptive weighted fusion strategy, as follows:

[0098] Fixed weights cannot adapt to changes in the environment. Visual weights are calculated based on lighting and image quality, using the following formula:

[0099] (18);

[0100] Based on point cloud quality and motion consistency, the specific formula is as follows:

[0101] (19);

[0102] in, This represents the normalized value of illumination. Indicates image blur. Represents the velocity variance. The standard deviation of the radial distance of the point cloud is used to measure the stability of the radar point cloud. The scale adjustment factor is typically set to 0.01. RCS is an equivalent area of ​​the target's ability to reflect electromagnetic waves under radar illumination, which is measured.

[0103] Step 3: High-precision state estimation and error compensation, as detailed below:

[0104] Lane geometry constraints are introduced to address distance errors caused by image perspective projection. The specific formula is as follows:

[0105] (20);

[0106] in, Indicates the angle between the vehicle and the lane line. Indicates the pitch angle (rad);

[0107] The Doppler spectrum broadening is solved through dominant frequency symmetry analysis, and the specific formula is as follows:

[0108] (twenty one);

[0109] in, Indicates spectral resolution. Represents the Doppler power spectrum;

[0110] Step 4: Output and quality assessment of the fusion results, as detailed below:

[0111] The time stamp, EMU coordinate system, vehicle speed, and confidence level are encoded into binary data and uploaded, while data quality health indicators are calculated:

[0112] (twenty two);

[0113] in, This represents the visual measurement state vector of the k-th target. This represents the radar measurement state vector of the k-th target;

[0114] when At that time, the multimodal fusion result is output normally;

[0115] when When this occurs, the online calibration process in step S1 is triggered;

[0116] when When this occurs, the visual channel is shut down, radar data is used only, and an alarm is triggered for recalibration.

[0117] Compared with the prior art, the present invention has the following advantages:

[0118] This invention enhances feature extraction capabilities by improving the YOLOP network architecture, replacing the traditional Bottleneck CSP module with a C2F module, and combining SPPF multi-scale pooling and coordinate attention mechanisms to improve the feature characterization of small targets, low-contrast traffic objects, and large vehicles. It also introduces ConvTranspose2D learnable upsampling technology to address the problem of blurred lane line segmentation edges. Addressing the pain point of large vehicle detection, it effectively alleviates class imbalance by synthesizing extreme scene data, dynamically adjusting class weights, adding dedicated anchor boxes, and employing a deformable RoL Pooling strategy, coupled with a Focal Loss optimization head with α=0.8 and γ=2, thus improving the recognition accuracy of occluded and deformed large vehicles. Furthermore, it constructs a spatial-channel dual-dimensional attention mechanism and a dynamic task allocation strategy, adaptively adjusting the sub-task loss weights based on real-time detection results to achieve collaborative optimization of target detection, lane segmentation, and ranging tasks. In the decision-level fusion stage, this invention breaks through the limitations of traditional geometric matching by constructing a multi-dimensional cost function that integrates geometry, motion, and semantics. It dynamically calculates modal weights based on illumination, image quality, and radar point cloud stability, achieving adaptive weighted fusion of visual and 77GHz FMCW radar data. Simultaneously, it reduces distance measurement and velocity estimation errors through lane geometry constraints and Doppler spectrum correction algorithms. This method does not rely on complex hardware upgrades; it achieves integrated execution of target recognition, lane segmentation, and traffic event monitoring solely through algorithm optimization. On edge devices, through hybrid precision optimization and model deployment strategies, it balances perception accuracy and real-time requirements, effectively solving the problems of unstable fusion accuracy, insufficient learning of difficult samples, and low edge deployment efficiency in existing technologies. It is applicable to various complex traffic scenarios such as urban roads, highways, and tunnels, providing high-precision and robust perception support for intelligent traffic management and vehicle-road cooperation. Attached Figure Description

[0119] Figure 1 Flowchart of a rapid traffic flow perception method based on multimodal fusion;

[0120] Figure 2 : Detection results of a traffic flow rapid perception method based on multimodal fusion;

[0121] Figure 3 Comparison of detection accuracy and speed of different traffic flow rapid perception methods for traffic objects. Detailed Implementation

[0122] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention:

[0123] The present invention describes a rapid traffic flow perception method based on multimodal fusion. This method constructs a perception network by deploying multi-source heterogeneous sensors such as cameras, millimeter-wave radar, and IMUs. Through a full-process processing including spatiotemporal alignment, edge preprocessing, improved network perception, decision-level fusion, global monitoring, and edge deployment, it achieves accurate and rapid perception of traffic flow. Compared with the shortcomings of existing traffic perception methods in multimodal data collaboration, small target / large vehicle detection accuracy, and edge real-time performance, this invention significantly improves the accuracy of traffic target detection and system response speed in complex scenarios by leveraging nanosecond-level synchronization, cross-layer feature fusion, a two-dimensional attention mechanism, and dedicated optimization strategies.

[0124] like Figure 1 As shown, this invention provides a method for rapid traffic flow perception based on multimodal fusion, comprising the following steps:

[0125] S1: Collaborative Data Acquisition and Spatiotemporal Alignment Using Multi-Source Heterogeneous Sensors:

[0126] Sensors are deployed using a layout of "radar centering and binocular visual angle". A nanosecond-level synchronization network based on PTPv2 protocol and GPS disciplined clock is constructed. Offline extrinsic parameter calibration is completed by minimizing reprojection error and SE(3) exponential mapping. In the online stage, the regenerating kernel Hilbert space (RKHS) similarity metric is used to achieve dynamic registration of visual features and radar signals. Combined with IMU pre-integration algorithm, pose error caused by road bumps is eliminated. The output is a spatiotemporally strictly aligned RGB image, semantic label and radar range-velocity spectrum data.

[0127] S2: Edge-side data preprocessing and feature-level fusion:

[0128] Data cleaning and standardization are performed in the edge computing unit, image noise is processed by median filtering, and distance correction is performed at the radar end. Primary trajectory prediction is performed based on Kalman filtering. The radar point cloud and visual detection box are initially matched through spatiotemporal correlation, and low-dimensional features containing target position and velocity are output, which effectively reduces data transmission bandwidth and improves system real-time performance.

[0129] S3: Multimodal target perception and feature extraction based on an improved network model:

[0130] An improved YOLOP network architecture is constructed: a C2F module and gradient splitting mechanism are introduced to enhance small target feature extraction, an SPPF module is used to implement adaptive multi-scale pooling, and ConvTranspose2D is used to optimize the upsampling process. A two-dimensional attention mechanism of coordinate attention (spatial) and ECA-Net (channel) is adopted to dynamically optimize feature weights. For the training process, Blender and GAN are used to generate extreme scene samples, and dynamic scale-aware adaptive focal loss (DSA-Focal Loss) is applied to improve detection robustness. At the same time, a dense correspondence between radar DBSCAN clustering and visual segmentation mask is established to achieve high-precision feature association.

[0131] S4: Multimodal decision-level fusion and accurate state recognition:

[0132] A multidimensional association cost function integrating IoU, optical flow velocity, and semantic similarity is constructed, and an adaptive weighted fusion strategy is implemented: visual weights are dynamically adjusted based on illumination and ambiguity, radar weights are calculated based on RCS (radar cross section) and the standard deviation of radial distance in point clouds, lane geometry constraint algorithm is introduced to correct visual perspective error, Doppler power spectrum main frequency symmetry analysis is used to optimize radar speed measurement, and finally the data quality health index (Q) is calculated. The fusion mode is dynamically switched or the online calibration process is triggered according to the score to ensure high confidence of the status output.

[0133] S5: Comprehensive Traffic Status Monitoring and Event Recognition

[0134] Based on the fusion results, full-segment monitoring is achieved: real-time statistics of traffic flow, speed, and trajectory; support for overspeed triggering warnings on illuminated vehicles and information boards; dedicated logic branches are designed for large vehicles to link active warnings, and events such as abnormal parking and low-speed driving are identified through reflection point density clustering and motion trajectory analysis, supporting traffic management and accident prevention;

[0135] S6: Model Optimization and Efficient Edge Deployment

[0136] During model training, class weights are dynamically adjusted, and targeted reinforcement is performed for large vehicles and difficult samples. During the deployment phase, the number of parameters is reduced through channel pruning. Combined with 8-bit quantization and TensorRT acceleration technology, low-latency real-time processing is achieved on edge computing platforms such as Jetson AGX Orin to meet the application needs of complex traffic scenarios.

[0137] Step S1 is as follows:

[0138] Step 1: Multi-sensor hardware deployment and synchronous triggering, as detailed below:

[0139] Cameras, millimeter-wave radar, IMU and other sensors are deployed at traffic monitoring nodes to form a perception area that covers 300 meters horizontally and 8 lanes vertically. The monitoring nodes are located on roadside pillars and gantries. The camera types include visible light or infrared, and the millimeter-wave radar can be 77GHz FMCW.

[0140] A sandwich layout with "radar centered and dual cameras at a 15° angle" is adopted. Geometric constraints ensure that the coverage of the overlapping area of ​​the field of view is >85%. The optimal installation angle is calculated based on the field of view angle formula.

[0141] (1);

[0142] in, Lane width, To monitor distance, For safety margin;

[0143] A synchronization network is constructed using the PTPv2 protocol, with a GPS-disciplined clock deployed as the master clock. Nanosecond-level synchronization is achieved through cable length compensation and spatial propagation delay compensation, as detailed in the following formula:

[0144] (2);

[0145] in, For sensor local timestamps, Where c is the length of the synchronization cable and c is the speed of light. ;

[0146] Step 2: Offline calibration and coordinate system one, as detailed below:

[0147] External parameter calibration:

[0148] The radar-camera extrinsic parameter matrix is ​​obtained using the checkerboard calibration method, and the reprojection error is minimized.

[0149] (3);

[0150] in, For Lie algebra parameters, For the SE(3) exponent mapping, For Huber robust kernel function, The regularization coefficient is used.

[0151] Coordinate system mapping:

[0152] With the camera's optical center as the origin of the world coordinate system, the radar coordinates (Xr, Zr) are directly mapped to the world coordinates (Xw, Zw). Yw is determined by measuring the radar's installation height using a laser rangefinder. The conversion formula is as follows:

[0153] (4);

[0154] Where h represents the radar installation height, addressing the issue of missing Y-axis data for the radar. ;

[0155] Step 3: Online dynamic registration and motion compensation, as detailed below:

[0156] Road sign registration:

[0157] Using fixed lane lines, road signs, and other features as baselines, a regenerative kernel Hilbert spatial similarity metric is constructed through visual recognition and radar reflection point matching.

[0158] (5);

[0159] in, For RBF kernel mapping functions, For feature space;

[0160] IMU tight coupling compensation:

[0161] Using the acceleration / angular velocity data collected by the IMU, the pose changes of the sensor platform caused by wind vibration or vehicle bumps are calculated, and the IMU data is pre-integrated.

[0162] (6);

[0163] in, This is the measured value of angular velocity (rad / s). Zero bias for the gyroscope (calibrated using Allan variance);

[0164] Step 4: Output data with spatiotemporal alignment, as detailed below:

[0165] Align the data from each sensor with a unified timestamp and output a spatiotemporally synchronized multimodal data packet, including visual, radar, and auxiliary data;

[0166] The visual data includes RGB images (1920×1080@30fps) and lane line semantic labels, the radar data includes target range-velocity spectrum (0.1m / 0.1km / h resolution), and the auxiliary data includes IMU pose parameters, road surface temperature or humidity.

[0167] Step S2 is as follows:

[0168] Data cleaning (threshold denoising), radar distance correction, image filtering (median / mean filtering), and format standardization are performed in the edge computing unit;

[0169] This method achieves primary trajectory prediction and fusion based on Kalman filtering. By using spatiotemporal correlation, radar point clouds are initially matched with visual detection boxes, and low-dimensional features such as target position and velocity are output, reducing the amount of data to be processed in the subsequent process and improving the real-time performance of the method.

[0170] Step S3 is as follows:

[0171] Backbone network optimization:

[0172] The original YOLOP's BottleneckCSP module is replaced by the C2F module, which enhances the feature extraction capability for small targets and low-contrast traffic objects through cross-layer feature fusion and channel stitching.

[0173] The C2F module introduces a gradient splitting mechanism to restrict the direct transmission of shallow features, prevent information redundancy, and improve model training stability. The specific formula is as follows:

[0174] (7);

[0175] in, Represents low-level high-resolution features. Represents high-level semantic features. This represents depthwise separable convolution;

[0176] Multi-scale feature enhancement:

[0177] The SPPF module (Fast Spatial Pyramid Pooling) is used to replace the traditional SPP module, and serial max pooling (kernel=5×5, 9×9, 13×13) is used to achieve adaptive multi-scale feature fusion, reducing the amount of computation by 30% while expanding the receptive field.

[0178] The addition of Coordinate Attention encodes the feature map in both horizontal and vertical directions, enhancing its sensitivity to the aspect ratio of vehicles and improving the detection accuracy of large vehicles.

[0179] High-efficiency upsampling design:

[0180] By replacing bilinear interpolation Upsample with ConvTranspose2D, the upsampling process is optimized through learnable parameters, avoiding the edge blurring problem of traditional interpolation methods and improving the continuity of lane lines in segmentation tasks.

[0181] Step 2: Two-dimensional attention mechanism and dynamic task allocation, as detailed below:

[0182] Spatial-channel coordinated attention:

[0183] Spatial dimension:

[0184] The coordinate attention module performs X / Y direction decomposition on the feature map to generate orientation-sensitive feature weights, as shown in the following formula:

[0185] Spacetime decomposition:

[0186] (8);

[0187] Spatiotemporal correlation modeling:

[0188] (9);

[0189] The intermediate attention response map after fusing spatiotemporal information;

[0190] Spatial weight generation:

[0191] (10);

[0192] in, Indicates the first Spatial location within each channel The relative importance weights of the corresponding features are normalized in the spatial dimension to enhance the response intensity of the vehicle body, edge contours, and interaction areas, while suppressing background interference information.

[0193] Channel dimension:

[0194] Efficient Channel Attention ECA-Net dynamically adjusts channel weights to highlight key features such as vehicle edges and textures while suppressing background noise. To avoid insufficient channel information caused by a single mean statistic, it introduces an efficient channel attention modeling method that integrates first-order and second-order statistics. The specific formula for channel statistics extraction is as follows:

[0195] Channel statistics extraction:

[0196] (11);

[0197] in, This represents the global statistical descriptor for the c-th channel; The second term represents the mean of the c-th channel and characterizes the dispersion of channel activation. To adjust the contribution ratio of mean and variance information, the hyperparameter was optimized to 0.4.

[0199] Adaptive convolution kernel calculation:

[0200] (12);

[0201] Dynamic gradient allocation for multiple tasks:

[0202] Design a differentiable task weight controller to dynamically adjust the loss function weights of target detection, lane segmentation, and ranging subtasks based on real-time detection results (e.g., reduce the segmentation task weight by 20% when a large vehicle is detected).

[0203] The detection head is optimized using Focal Loss (α=0.8, γ=2) to enhance learning on difficult samples such as those with occlusion and small vehicles. The proposed dynamic scale-aware adaptive focus loss DSA-Focal Loss is also employed to optimize the detection head.

[0204] (13);

[0205] Here, α is 0.8, used to adjust the relative contribution of positive and negative samples to the total loss. To predict class probabilities, γ is set to 2, which is used to suppress the loss contribution of easily classified samples and enhance the model's robustness to targets in occluded, small-scale, and complex traffic scenarios. The value is 0.5, which is the scale enhancement coefficient used to adjust the degree of weight amplification for small-scale targets. This represents the effective receptive region area of ​​the target on the feature map, used to characterize the target's scale. , which is the scale normalization reference constant, is taken as the average value of all receptive regions of this category;

[0206] Step 3: Specific optimization strategies for large vehicles, as follows:

[0207] Data augmentation and sample balancing:

[0208] Blender was used to generate synthetic data for extreme scenarios such as container trucks in snowy weather and oil tankers in foggy weather. The GAN network generated partially occluded and deformed samples, and the proportion of large vehicle training samples was increased to 30%.

[0209] Dynamic category weight adjustment:

[0210] When the detection rate of large vehicles is below the threshold, their loss weight is automatically increased (up to 300%) to resolve the class imbalance problem.

[0211] (14);

[0212] in, This represents the loss weight corresponding to the large vehicle in the t-th training period; This indicates the weight of large vehicle losses in the previous period. This is the time smoothing coefficient, used to control the degree of influence of historical weights on the current update, and is set to 0.1; This is the difficulty amplification factor, used to adjust the extent to which the detection difficulty increases the weight of the loss; it is set to 0.4. Let be the recall rate of large vehicles in the t-th training period. The average recall rate across all categories;

[0213] Structural adaptability improvements:

[0214] A length-to-width ratio sensitive anchor frame (8:1, 5:1) is added to the detection head to match special vehicle types such as trucks and trailers;

[0215] For large vehicles, a high-resolution feature map branch is used, and shallow detail features are fused using the C2F module to avoid the loss of small-scale features;

[0216] Deformable Rol Pooling:

[0217] To address the feature misalignment caused by fixed-grid pooling, a sub-network is used to predict the sampling point offset. The specific formula is as follows:

[0218] (15);

[0219] in, This represents the coordinates of the original sampling points; FC represents a fully connected layer.

[0220] Multimodal contour constraints:

[0221] To address the inconsistency between radar and visual contours, a dense correspondence between radar point clouds and visual segmentation masks is established, with the specific formula as follows:

[0222] (16);

[0223] in, This represents the DBSCAN clustering results;

[0224] Step 4: Real-time feature output and intermediate results, as detailed below:

[0225] The detection targets include vehicle location (BBox), type, and confidence level, where vehicle type includes cars or trucks, etc.

[0226] Semantic segmentation is performed on lane lines and drivable area masks, where the drivable area mask resolution is 512×512.

[0227] The vertical distance is output based on inverse perspective transformation (IPM) of monocular vision, where the vertical distance error must be less than 5%@50m;

[0228] Output cross-layer feature fusion map and SPPF multi-scale feature map from the C2F module for model interpretability analysis.

[0229] Step S4 is as follows:

[0230] Step 1: Cross-modal target association and confidence assessment, as detailed below:

[0231] Traditional methods only consider geometric location matching. To achieve multi-dimensional matching cost calculation, a cost function that integrates geometry, motion, and semantics is constructed. The specific formula is as follows:

[0232] (17);

[0233] in, Indicates the intersection and union of 3D bounding boxes. Indicates the visual optical flow estimation velocity. Indicates radar Doppler velocity, Indicates semantic similarity score;

[0234] Step 2: Adaptive weighted fusion strategy, as follows:

[0235] Fixed weights cannot adapt to changes in the environment. Visual weights are calculated based on lighting and image quality, using the following formula:

[0236] (18);

[0237] Based on point cloud quality and motion consistency, the specific formula is as follows:

[0238] (19);

[0239] in, This represents the normalized value of illumination. Indicates image blur. Represents the velocity variance. The standard deviation of the radial distance of the point cloud is used to measure the stability of the radar point cloud. The scale adjustment factor is typically set to 0.01. RCS is an equivalent area of ​​the target's ability to reflect electromagnetic waves under radar illumination, which is measured.

[0240] Step 3: High-precision state estimation and error compensation, as detailed below:

[0241] Lane geometry constraints are introduced to address distance errors caused by image perspective projection. The specific formula is as follows:

[0242] (20);

[0243] in, Indicates the angle between the vehicle and the lane line. Indicates the pitch angle (rad);

[0244] The Doppler spectrum broadening is solved through dominant frequency symmetry analysis, and the specific formula is as follows:

[0245] (twenty one);

[0246] in, Indicates spectral resolution. Represents the Doppler power spectrum;

[0247] Step 4: Output and quality assessment of the fusion results, as detailed below:

[0248] The time stamp, EMU coordinate system, vehicle speed, and confidence level are encoded into binary data and uploaded, while data quality health indicators are calculated:

[0249] (twenty two);

[0250] in, This represents the visual measurement state vector of the k-th target. This represents the radar measurement state vector of the k-th target;

[0251] when At that time, the multimodal fusion result is output normally;

[0252] when When this occurs, the online calibration process in step S1 is triggered;

[0253] when When this occurs, the visual channel is shut down, radar data is used only, and an alarm is triggered for recalibration.

[0254] Step S5 is as follows:

[0255] Full-segment monitoring based on fusion results:

[0256] 1) Real-time statistics on traffic, speed, and trajectory;

[0257] 2) Overspeed (adjustable threshold) triggers warnings on the lighting vehicle and information board;

[0258] 3) Active early warning linked to dedicated logic branches for large vehicles.

[0259] By using reflection point density clustering and motion trajectory analysis, events such as abnormal parking and low-speed driving can be identified, supporting traffic management and accident prevention.

[0260] Step S6 is as follows:

[0261] Extreme weather and occlusion samples are generated using the Blender physics engine and GAN; class weights are dynamically adjusted during training (the loss weight for large vehicles is increased by up to 300%), and Focal Loss is used to reinforce difficult samples;

[0262] During deployment, channel pruning (reducing the number of parameters), 8-bit quantization, and TensorRT acceleration are used to achieve real-time processing on the Jetson AGXOrin platform, meeting the low latency requirements of the edge.

[0263] This invention proposes a rapid traffic flow perception method based on multimodal fusion, which mainly includes collaborative data acquisition and spatiotemporal alignment from multi-source heterogeneous sensors, edge-side data preprocessing and feature-level fusion, multimodal target perception and feature extraction based on an improved network model, multimodal decision-level fusion and accurate state recognition, full-domain traffic state monitoring and event recognition, model optimization and efficient edge deployment.

[0264] Case 1: For example Figure 2 As shown in the diagram, the traffic target detection effect of the method described in this invention patent is illustrated. The experimental results show that the method can accurately identify 12 types of vehicles within a 300-meter detection range. It can clearly capture the outlines of special vehicle types such as large container trucks and tank trucks. It has high accuracy in lane line segmentation and vehicle positioning. Even with slight occlusion, it can effectively distinguish targets. This provides intuitive support for verifying the multimodal fusion detection accuracy and adaptability to complex scenarios of the method described in this invention.

[0265] Case 2: For example Figure 3 As shown, the comparison data of detection accuracy and speed of different methods for traffic objects reveals that, compared to the YOLOv8 model, the method of this invention improves mAP@0.5 on both the KITTI and VOC datasets, reduces the number of parameters by more than 27.8%, and increases inference speed by 1.56 to 1.90 times, demonstrating superior performance balance. This data provides quantitative evidence to verify the comprehensive superiority of the method of this invention in terms of accuracy, lightweight design, and real-time performance, as well as its applicability for edge deployment.

[0266] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for rapid traffic flow perception based on multimodal fusion, characterized in that, Includes the following steps: S1: Multi-source heterogeneous sensor collaborative data acquisition and spatiotemporal alignment: Sensors are deployed in a layout of "radar centering and binocular vision angle", a nanosecond-level synchronization network based on PTPv2 protocol and GPS disciplined clock is constructed, and offline extrinsic parameter calibration is completed by minimizing reprojection error and SE(3) exponential mapping. In the online stage, the regenerating kernel Hilbert spatial similarity metric is used to realize the dynamic registration of visual features and radar signals. Combined with IMU pre-integration algorithm, pose error caused by road bumps is eliminated, and spatiotemporally strictly aligned RGB images, semantic labels and radar range-velocity spectrum data are output. S2: Edge-side data preprocessing and feature-level fusion: Data cleaning and standardization are performed in the edge computing unit, image noise is processed by median filtering, and distance correction is performed at the radar end. Primary trajectory prediction is performed based on Kalman filtering. The radar point cloud and visual detection box are initially matched through spatiotemporal correlation, and low-dimensional features containing target position and velocity are output. S3: Multimodal target perception and feature extraction based on an improved network model: This includes constructing an improved YOLOP network architecture: introducing a C2F module and gradient splitting mechanism to enhance small target feature extraction, utilizing the SPPF module to achieve adaptive multi-scale pooling, and using ConvTranspose2D to optimize the upsampling process. It also employs a dual-dimensional attention mechanism of coordinate attention and ECA-Net to dynamically optimize feature weights. For the training process, Blender and GAN are used to generate extreme scene samples, and dynamic scale-aware adaptive focus loss is applied to improve detection robustness. Simultaneously, a dense correspondence is established between radar DBSCAN clustering and visual segmentation masks to achieve high-precision feature association. S4: Multimodal decision-level fusion and accurate state recognition: The system includes constructing a multi-dimensional association cost function that integrates IoU, optical flow velocity, and semantic similarity, and implementing an adaptive weighted fusion strategy: visual weights are dynamically adjusted based on illumination and ambiguity, radar weights are calculated based on RCS and the standard deviation of radial distance in point clouds, lane geometry constraint algorithms are introduced to correct visual perspective errors, Doppler power spectrum frequency symmetry analysis is used to optimize radar speed measurement, and finally, data quality health indicators are calculated. The fusion mode is dynamically switched or an online calibration process is triggered based on the score to ensure high confidence in the status output. S5: Comprehensive Traffic Status Monitoring and Event Recognition Includes full-segment monitoring based on fusion results: real-time statistics of traffic flow, speed, and trajectory; supports overspeed triggering warnings from lighting vehicles and information boards; A dedicated logic branch is designed for large vehicles to trigger proactive early warnings. Through reflection point density clustering and motion trajectory analysis, events including abnormal parking and low-speed driving are identified to support traffic management and accident prevention. S6: Model Optimization and Efficient Edge Deployment During model training, class weights are dynamically adjusted, and targeted reinforcement is performed for large vehicles and difficult samples. During the deployment phase, the number of parameters is reduced through channel pruning. Combined with 8-bit quantization and TensorRT acceleration technology, low-latency real-time processing is achieved on edge computing platforms such as Jetson AGX Orin to meet the application needs of complex traffic scenarios.

2. The method for rapid traffic flow perception based on multimodal fusion according to claim 1, characterized in that, Step S1, multi-source heterogeneous sensor collaborative data acquisition and spatiotemporal alignment, specifically includes the following steps: Step 1: Multi-sensor hardware deployment and synchronous triggering, as detailed below: Sensors are deployed at traffic monitoring nodes to form a sensing area. The locations of the monitoring nodes include roadside pillars and gantries. The sensors include cameras, millimeter-wave radar, and IMU. The camera types include visible light or infrared, and the millimeter-wave radar includes 77GHz FMCW. A sandwich layout with "radar centered and dual cameras at a 15° angle" is adopted. Geometric constraints ensure that the coverage of the overlapping area of ​​the field of view is >85%. The installation angle is calculated based on the field of view angle formula. （1）； in, Lane width, To monitor distance, For safety margin; A synchronization network is constructed using the PTPv2 protocol, with a GPS-disciplined clock deployed as the master clock. Nanosecond-level synchronization is achieved through cable length compensation and spatial propagation delay compensation, as detailed in the following formula: （2）； in, For sensor local timestamps, Where c is the length of the synchronization cable and c is the speed of light. ; Step 2: Offline calibration and coordinate system one, as detailed below: The radar-camera extrinsic parameter matrix is ​​obtained using the checkerboard calibration method, and the reprojection error is minimized. （3）； in, For Lie algebra parameters, For the SE(3) exponent mapping, For Huber robust kernel function, The regularization coefficient is used. With the camera's optical center as the origin of the world coordinate system, the radar coordinates (Xr, Zr) are directly mapped to the world coordinates (Xw, Zw). Yw is determined by measuring the radar's installation height using a laser rangefinder. The conversion formula is as follows: （4）； Where h is the radar installation height, ; Step 3: Online dynamic registration and motion compensation, as detailed below: Using fixed lane lines, road signs, and other features as baselines, a regenerative kernel Hilbert spatial similarity metric is constructed through visual recognition and radar reflection point matching. （5）； in, For RBF kernel mapping functions, For feature space; This represents the structured feature vector extracted by the visual sensor, including but not limited to lane line geometric parameters, target bounding box center coordinates, and scale information; This represents the target feature vector extracted by millimeter-wave radar, including the target's radial range, radial velocity, Doppler intensity, and azimuth information. Using the acceleration / angular velocity data collected by the IMU, the pose changes of the sensor platform caused by wind vibration or vehicle bumps are calculated, and the IMU data is pre-integrated. （6）； in, This is the measured value of angular velocity. Zero bias for the gyroscope; The total number of samples taken by the IMU within a time synchronization window; This is the discrete sampling index for the IMU, used to represent the k-th time sampling point; The time interval between two adjacent IMU samples; Step 4: Output data with spatiotemporal alignment, as detailed below: Align the data from each sensor with a unified timestamp and output a spatiotemporally synchronized multimodal data packet, including visual, radar, and auxiliary data; The visual data includes RGB images and lane line semantic labels, the radar data includes target distance-velocity spectrum, and the auxiliary data includes IMU pose parameters, road surface temperature or humidity.

3. The method for rapid traffic flow perception based on multimodal fusion according to claim 1, characterized in that, Step S3, based on the improved network model for multimodal target perception and feature extraction, specifically includes the following steps: Step 1: Design of a multi-scale feature fusion network architecture, as follows: The original YOLOP's BottleneckCSP module is replaced by the C2F module, which enhances the feature extraction capability for small targets and low-contrast traffic objects through cross-layer feature fusion and channel stitching. The C2F module introduces a gradient-based flow splitting mechanism, with the specific formula as follows: （7）； in, Represents low-level high-resolution features. Represents high-level semantic features. This represents depthwise separable convolution; Intermediate layer feature maps are used to supplement the transition information between scale and semantics; The traditional SPP module is replaced by the fast spatial pyramid pooling (SPPF) module, and adaptive multi-scale feature fusion is achieved by serial max pooling. A new Coordinate Attention feature is added to encode the feature map in the horizontal and vertical directions. Replace bilinear interpolation Upsample with ConvTranspose2D and optimize the upsampling process using learnable parameters; Step 2: Two-dimensional attention mechanism and dynamic task allocation, as detailed below: The coordinate attention module performs X / Y direction decomposition on the feature map to generate orientation-sensitive feature weights, as shown in the following formula: Spacetime decomposition: （8）； Input a feature tensor, where T represents the time dimension, H and W represent the spatial height and width, and C represents the number of channels; The feature representation obtained by aggregating along the spatial height direction is used to model horizontal spatial correlation; The feature representation obtained by aggregating along the spatial width direction is used to model vertical spatial correlation; The feature representation obtained by aggregating along the time dimension is used to model cross-frame temporal consistency.

4. Spatiotemporal correlation modeling: （9）； in, The intermediate attention response map after fusing spatiotemporal information; Spatial weight generation: （10）； in, Indicates the first Spatial location within each channel The relative importance weights of the corresponding features are normalized in the spatial dimension to enhance the response intensity of the vehicle body, edge contours, and interaction areas, while suppressing background interference information. Efficient Channel Attention ECA-Net dynamically adjusts channel weights to highlight key features including vehicle edges and textures. Meanwhile, to avoid insufficient channel information caused by a single mean statistic, an efficient channel attention modeling method fusing first-order and second-order statistics is introduced. The specific formula for channel statistics extraction is as follows: （11）； in, This represents the global statistical descriptor for the c-th channel; The second term represents the mean of the c-th channel and characterizes the dispersion of channel activation. To adjust the contribution ratio of mean and variance information, the hyperparameter was optimized to 0.

4. Adaptive convolution kernel calculation: （12）； This is the channel attention weight vector, used to highlight key semantics and edge features; Design a differentiable task weight controller to dynamically adjust the loss function weights of the target detection, lane segmentation, and ranging subtasks based on real-time detection results; The proposed dynamic scale-aware adaptive focus loss DSA-Focal Loss optimization detection head is employed: （13）； Here, α is 0.8, used to adjust the relative contribution of positive and negative samples to the total loss. To predict class probabilities, γ is set to 2, which is used to suppress the loss contribution of easily classified samples and enhance the model's robustness to targets in occluded, small-scale, and complex traffic scenarios. The value is 0.5, which is the scale enhancement coefficient used to adjust the degree of weight amplification for small-scale targets. This represents the effective receptive region area of ​​the target on the feature map, used to characterize the target's scale. , which is the scale normalization reference constant, is taken as the average value of all receptive regions of this category; Step 3: Specific optimization strategies for large vehicles, as follows: Blender was used to generate synthetic data for extreme scenarios such as container trucks in snowy weather and oil tankers in foggy weather. The GAN network generated partially occluded and deformed samples, and the proportion of large vehicle training samples was increased to 30%. When the detection rate of large vehicles is below the threshold, their loss weight is automatically increased; （14）； in, This represents the loss weight corresponding to the large vehicle in the t-th training period; This indicates the weight of large vehicle losses in the previous period. This is the time smoothing coefficient, used to control the degree of influence of historical weights on the current update, and is set to 0.1; This is the difficulty amplification factor, used to adjust the extent to which the detection difficulty increases the weight of the loss; it is set to 0.

4. Let be the recall rate of large vehicles in the t-th training period. The average recall rate across all categories; A dimension-sensitive anchor frame is added to the detection head to match special vehicle types such as trucks and trailers; For large vehicles, a high-resolution feature map branch is used, and shallow detail features are fused using the C2F module to avoid the loss of small-scale features; The sampling point offset is predicted using a subnetwork, and the specific formula is as follows: （15）； in, Represents the two-dimensional coordinates of the original sampling points. This represents the coordinates of the sampling point after offset correction; Indicates the predicted offset of the sampling point; FC indicates a fully connected layer; This represents the feature representation of the corresponding candidate target region (ROI); A dense correspondence is established between radar point clouds and visual segmentation masks, using the following formula: （16）； in, This represents the DBSCAN clustering results; Step 4: Real-time feature output and intermediate results, as detailed below: The detection targets include vehicle location, type, and confidence level, where vehicle type includes cars or trucks, etc. Semantic segmentation is performed on lane lines and drivable area masks, where the drivable area mask resolution is 512×512. The longitudinal distance is output based on the inverse perspective transformation of monocular vision, where the longitudinal distance error must be less than 5%@50m; Output the cross-layer feature fusion map of the C2F module and the SPPF multi-scale feature map.

5. The method for rapid traffic flow perception based on multimodal fusion according to claim 1, characterized in that, Step S4, multimodal decision-level fusion and accurate state recognition, specifically includes the following steps: Step 1: Cross-modal target association and confidence assessment, as detailed below: A cost function integrating geometry, motion, and semantics is constructed, and the specific formula is as follows: （17）； in, Indicates the intersection and union of 3D bounding boxes. Indicates the visual optical flow estimation velocity. Indicates radar Doppler velocity, Indicates semantic similarity score; Step 2: Adaptive weighted fusion strategy, as follows: The visual weights are calculated based on illumination and image quality, using the following formula: （18）； Based on point cloud quality and motion consistency, the specific formula is as follows: （19）； in, This represents the normalized value of illumination. Indicates image blur. Represents the velocity variance. The standard deviation of the radial distance of the point cloud is used to measure the stability of the radar point cloud. is the scale adjustment factor; RCS is an equivalent area of ​​the target's ability to reflect electromagnetic waves under radar illumination, which is measured. Step 3: High-precision state estimation and error compensation, as detailed below: Lane geometry constraints are introduced to address distance errors caused by image perspective projection. The specific formula is as follows: （20）； in, Indicates the angle between the vehicle and the lane line. Indicates the pitch angle (rad); The key frequency symmetry analysis is presented using the following formula: （21）； in, Indicates spectral resolution. Represents the Doppler power spectrum; Step 4: Output and quality assessment of the fusion results, as detailed below: The time stamp, EMU coordinate system, vehicle speed, and confidence level are encoded into binary data and uploaded, while data quality health indicators are calculated: （22）； in, This represents the visual measurement state vector of the k-th target. This represents the radar measurement state vector of the k-th target; when At that time, the multimodal fusion result is output normally; when When this occurs, the online calibration process in step S1 is triggered; when When this occurs, the visual channel is shut down, radar data is used only, and an alarm is triggered for recalibration.

Images