Complex environment traffic parameter detection method and system based on laser radar and vision fusion

Abstract

The invention belongs to the field of intelligent traffic systems, and particularly discloses a complex environment traffic parameter detection method and system based on laser radar and visual fusion, and the system comprises a multi-modal data collection and preprocessing module, a dynamic weight fusion module, and a multi-modal fusion feature map feature extraction and traffic parameter calculation module. The method is suitable for a complex environment (such as severe weather and illumination variation), realizes high-precision and real-time acquisition of traffic flow parameters in the complex environment, and provides core data support for intelligent traffic signal control and road network optimization.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation systems, and in particular to a method and system for detecting traffic parameters in complex environments based on the fusion of lidar and vision. Background Technology

[0002] Traffic flow parameter detection is a core foundation for intelligent traffic signal control, road network planning, and travel guidance. Current technologies for traffic flow detection primarily rely on single sensors or simple fusion schemes, which have the following limitations: (1) Poor environmental adaptability of single sensors Visual sensors (cameras): rely on ambient light; detection accuracy drops by more than 50% in rain, snow, fog, and backlighting scenarios, and vehicle classification accuracy is less than 65%. LiDAR (Light Detection and Ranging): It lacks semantic information in 3D point clouds (cannot directly identify vehicle models and lane lines), and the data volume of a single point cloud is large (100,000 points / frame), making real-time processing difficult; Millimeter-wave radar: low resolution (difficult to distinguish adjacent vehicles), susceptible to interference from reflections from metallic objects.

[0003] (2) Staticization of traditional fusion methods Existing fusion technologies mostly use fixed weights (such as a 7:3 ratio of LiDAR to vision weights), without considering dynamic changes in the environment (such as the need to increase the radar weights when going from sunny to rainy days). This results in insufficient robustness in complex scenarios, with detection accuracy fluctuating by more than 20%.

[0004] (3) Insufficient real-time performance at the edge Mainstream algorithms rely on cloud computing (such as Faster R-CNN and PointNet), with model parameters exceeding 100M. The inference latency of edge devices (such as roadside units) is >200ms, which cannot meet the real-time requirements of traffic signal control (which requires a latency of ≤100ms).

[0005] Therefore, there is an urgent need for a dynamic, environmentally adaptive, lightweight, and high-precision multi-sensor fusion detection method to solve the technical bottleneck of traffic flow parameter acquisition in complex scenarios. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for detecting traffic parameters in complex environments based on the fusion of lidar and vision.

[0007] To achieve the above objectives, the present invention is implemented according to the following technical solution: The first technical solution provided by this invention is a method for detecting traffic parameters in complex environments based on the fusion of lidar and vision, comprising the following steps: S1. Install a LiDAR and a camera, calibrated using the checkerboard joint calibration method, coaxially above the traffic intersection, and simultaneously acquire the LiDAR point cloud data collected by the LiDAR and the visual image data collected by the camera. S2. Preprocess the lidar point cloud data and visual image data respectively; S3. Project the preprocessed LiDAR point cloud onto the visual image plane to generate two-dimensional pixel coordinates, and generate a radar heatmap through reflection intensity and density encoding; convert the three-dimensional point cloud into a two-dimensional feature map aligned with the visual image space to highlight the vehicle target area; calculate the LiDAR point cloud density and reflection intensity variance, and obtain the LiDAR point cloud density confidence score and reflection intensity confidence score after normalizing the LiDAR point cloud density and reflection intensity confidence score; combine the LiDAR point cloud density confidence score and reflection intensity confidence score to obtain the LiDAR point cloud comprehensive confidence score. S4. Classify the preprocessed visual image by environment type and environment recognition factor according to the preset environment type and output the corresponding environment factor; S5. The LiDAR weight is obtained by combining the comprehensive confidence of the LiDAR point cloud and environmental factors. The visual weight is obtained based on the characteristic that the sum of the LiDAR weight and the visual weight is equal to 1. S6. Superimpose the radar heat map and the visual image according to the weights to generate a fused feature map, and perform normalization and Gaussian smoothing on the fused feature map to obtain a multimodal fused feature map. S7. Input the multimodal fusion feature map into the improved MobileNetV3 network model for feature extraction. The features include traffic flow, vehicle speed, queue length and vehicle type classification features. Calculate the traffic flow, vehicle speed, queue length and vehicle type classification of the traffic intersection based on the extracted traffic flow, vehicle speed, queue length and vehicle type classification features.

[0008] Furthermore, in step S1, the PPS second pulse signal is output through the GPS timing module to synchronously trigger the lidar and camera to collect lidar point cloud data and visual image data, with a timestamp accuracy of ≤1μs.

[0009] Furthermore, in step S2, the preprocessing of the LiDAR point cloud data includes denoising, downsampling, and ROI cropping; the preprocessing of the visual image data includes distortion correction and enhancement and visual feature alignment. Visual feature alignment is achieved by extracting lane lines and vehicle edge features from the visual image through Canny edge detection to ensure that the visual image is spatially aligned with the vehicle target area in the LiDAR heatmap.

[0010] Furthermore, in step S3, the formula for calculating the overall confidence level of the lidar point cloud is: ; In the formula: This represents the overall confidence level of the lidar point cloud, ranging from [0,1]. This indicates the confidence level of the lidar point cloud density. If the lidar point cloud density is >100 points / m², then... 2 , If the point cloud density of the lidar is less than 30 points / m 2 , ; This represents the confidence level of the lidar point cloud reflection intensity. If the variance of the lidar point cloud reflection intensity is less than 50, If the variance of the reflectance intensity of the lidar point cloud is > 200, .

[0011] Furthermore, in step S4, the preset environment types and environment identification factors are as follows: .

[0012] Furthermore, in step S5, the formula for calculating the lidar weight is: ; In the formula: Indicates the weight of the lidar; This represents the overall confidence level of the lidar point cloud, ranging from [0,1]. Indicates environmental factors; This represents the balance coefficient, with a value of 0.4.

[0013] Furthermore, in step S6, the formula for generating the fused feature map is: ; In the formula: Represents the fused feature map; Indicates the weight of the lidar; A three-channel pseudo-color image representing a lidar heat map, with pixel values ​​ranging from 0 to 255; Indicates visual weight, ; A three-channel RGB image representing a visual image, with pixel values ​​ranging from 0 to 255; This represents the two-dimensional pixel coordinates generated by projecting the preprocessed LiDAR point cloud onto the visual image plane. This represents the RGB channels (0:R, 1:G, 2:B).

[0014] Furthermore, in step S7: Traffic flow: Based on the detection of target boxes using YOLOv5s lightweight version, combined with SORT algorithm trajectory tracking, the count is based on the entrance lane, vehicles / 5 minutes; Vehicle speed: Calculated by the displacement of the target center pixel in adjacent frames, formula: ;in Z The distance to the target vehicle measured by lidar. f x For camera focal length, fps For frame rate; Queue length: DBSCAN clusters point cloud vehicle clusters and calculates the distance between the first and last points, in meters; Vehicle type classification: Based on the length / width / height of the LiDAR point cloud and the aspect ratio of the visual image, vehicles are classified into small vehicles (length < 4.5m), medium-sized vehicles (length 4.5-6m), and large vehicles (length > 6m).

[0015] The second technical solution provided by this invention is a complex environment traffic parameter detection system based on the fusion of lidar and vision, comprising: The multimodal data acquisition and preprocessing module includes a LiDAR and a camera coaxially mounted above a traffic intersection and calibrated using a checkerboard joint calibration method; a GPS timing module for outputting PPS (pixel per second) signals to synchronously trigger the LiDAR and camera to acquire LiDAR point cloud data and visual image data; a LiDAR point cloud data preprocessing module for denoising, downsampling, and ROI cropping of the LiDAR point cloud data; and a visual image data preprocessing module for distortion correction, enhancement, and visual feature alignment of the visual image data. The dynamic weight fusion module includes an environment recognition module for classifying the preprocessed visual image into environmental types and outputting corresponding environmental factors; a dynamic weight calculation module for calculating the LiDAR weight and the visual weight; and a weight superposition fusion module for superimposing the radar heatmap and the visual image according to weights to generate a fusion feature map, and performing normalization and Gaussian smoothing on the fusion feature map to obtain a multimodal fusion feature map. The multimodal fusion feature map feature extraction and traffic parameter calculation module includes a MobileNetV3 network model feature extraction module for extracting traffic flow, vehicle speed, queue length and vehicle type classification features from the multimodal fusion feature map; and a traffic flow parameter calculation module for calculating traffic flow, vehicle speed, queue length and vehicle type classification at traffic intersections.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Enhanced Environmental Adaptability: The fused data covers both semantic and geometric dimensions, improving the recognition accuracy of traffic flow, vehicle speed, and vehicle type by more than 20% compared to single sensors. Detection accuracy in rainy / foggy scenes is 40% higher than pure vision, with traffic flow (MAE = 5 vehicles / hour, vehicle speed RMSE = 3km / h); vehicle type classification accuracy in backlight scenes is ≥88% (pure vision only 65%).

[0017] 2. Real-time performance at the edge meets the requirements: Model size 8.5MB, inference latency ≤80ms, meeting the real-time requirements of roadside equipment (frame rate ≥25fps).

[0018] 3. Dynamic weight robustness: Effectively overcomes environmental interference such as low light, rain and snow, and the detection accuracy remains above 85% even in adverse weather conditions; when the environment changes (such as sunny day to rainy day), the weight adjustment response time is ≤100ms and the accuracy fluctuation is ≤5%. Attached Figure Description

[0019] Figure 1 This is a structural block diagram of the complex environment traffic parameter detection system based on the fusion of lidar and vision according to the present invention.

[0020] Figure 2 This is a flowchart of the dynamic weight fusion algorithm. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0022] like Figure 1 As shown, this embodiment first provides a traffic parameter detection system for complex environments based on the fusion of lidar and vision, including: The multimodal data acquisition and preprocessing module includes a LiDAR and a camera coaxially mounted above a traffic intersection and calibrated using a checkerboard joint calibration method; a GPS timing module for outputting PPS (pixel per second) signals to synchronously trigger the LiDAR and camera to acquire LiDAR point cloud data and visual image data; a LiDAR point cloud data preprocessing module for denoising, downsampling, and ROI cropping of the LiDAR point cloud data; and a visual image data preprocessing module for distortion correction, enhancement, and visual feature alignment of the visual image data. The lidar used is a 32-line mechanical lidar (such as the Hesai PandarXT), with the following parameters: horizontal field of view 120°, vertical field of view -10° to +20°, frame rate 10Hz; ranging range 0.5-200m (error ≤ ±3cm), point cloud density ≥ 100 points / m². 2@50m; Supports PPS triggering (timestamp accuracy ≤1μs), outputs point cloud including coordinates (X / Y / Z) and reflection intensity; Camera: 2-megapixel industrial camera (e.g., Dahua DH-IPC-HFW5249T-ZE), parameters meet: resolution 1920×1080, frame rate 25Hz, focal length 6mm (field of view 60°×45°); Wide dynamic range (WDR) ≥120dB, supports PPS triggering (timestamp accuracy ≤10μs); Distortion coefficients: radial distortion ≤1%, tangential distortion ≤0.5%; Outputs PPS second pulse signal through GPS timing module to synchronously trigger LiDAR (10Hz) and camera (25Hz) acquisition, timestamp accuracy ≤1μs, achieving time synchronization between LiDAR and camera; Obtains extrinsic parameter matrix (rotation matrix R, translation vector T) through checkerboard calibration board (Zhang's calibration method) to achieve spatial coordinate transformation (point cloud → image pixel coordinate system). It should be noted that the checkerboard grid must be rectangular, not square, otherwise the calibration tool will not be able to determine the direction. The recommended size is 50cm × 42cm, but it can be adjusted according to actual needs. The checkerboard grid should be as large as possible so that the LiDAR scanning will be more accurate. One side of the checkerboard grid should have an odd number of squares and the other side should have an even number of squares. The black and white squares should have different reflectivities, which can be used for both camera calibration and LiDAR calibration. There should be one odd and one even black square at each end of the checkerboard grid, and the squares should not be too small. Calibration should be performed in an indoor laboratory with a stable light source, avoiding direct sunlight and shadow interference. Ensure the calibration board surface is flat, without wrinkles or deformation. The LiDAR and camera should be mounted on a rigid platform to ensure their relative positions remain constant. The equipment must not move during calibration to maintain stability. The checkerboard pattern should be placed within the common field of view of the LiDAR and camera. The checkerboard pattern needs to be at different angles to the LiDAR and camera; it is recommended to collect data from multiple orientations. Orientation requirements: the short side of the checkerboard pattern should be at the top of the image; the edge of the checkerboard pattern should not be parallel to the edge of the image; in the point cloud, the short side of the checkerboard pattern should be at the top of the LiDAR along the z-axis; the edge of the checkerboard pattern in the point cloud should not be parallel to the z-axis of the LiDAR. The calibration process typically includes the following steps: 1) Data acquisition: The GPS timing module outputs a PPS second pulse signal to synchronously trigger the lidar (10Hz) and camera (25Hz) to acquire lidar point cloud data and visual image data. 2) Feature extraction: After extracting corner points or features from the visual image, extract the corresponding 3D points from the LiDAR point cloud; 3) Feature matching: Matching feature points in the visual image with 3D points in the LiDAR point cloud; 4) External parameter solution: Use optimization algorithms (such as least squares method) to calculate the rotation matrix and translation vector; The transformation matrix from the lidar coordinate system (LCS) to the camera coordinate system (CCS) is solved using the checkerboard joint calibration method. ; Where R is the rotation matrix (3×3) and T is the translation vector (3×1). Optimized through 20 sets of calibration samples (chessboard corner points), the reprojection error is ≤0.5 pixels. The lidar coordinate system (LCS) has the origin at the lidar center, the X-axis pointing forward (towards the detection area), the Y-axis pointing left, and the Z-axis pointing upward (right-hand coordinate system). These are the coordinates of the point cloud in the lidar coordinate system.

[0023] Visual camera coordinate system (CCS): The origin is the optical center of the camera, the X-axis is to the right (image column direction), the Y-axis is downward (image row direction), and the Z-axis is forward along the optical axis (consistent with the camera focal length direction). The coordinates of the point cloud are in the visual camera coordinate system.

[0024] The pixel coordinates of the point cloud on the image are obtained by projecting the camera intrinsic parameters onto the image pixel coordinate system (UCS).

[0025] ; in: These are the pixel coordinates of the image; The camera focal length (in pixels); The coordinates of the camera's principal point (the center pixel of the image).

[0026] 5) Error assessment and optimization: assess the accuracy of the calibration results and optimize them if necessary.

[0027] The lidar point cloud preprocessing module is involved in: Noise reduction: Statistical outlier removal is used to remove outliers whose distance from the mean is greater than 2 standard deviations. In rainy weather, outliers with a reflectance intensity < 200 (low-reflectance interference points) are removed. Downsampling: Voxel grid filtering (voxel size 0.1m×0.1m×0.1m), point cloud count reduced from 100,000 to 40,000 points / frame; ROI clipping: Preserve the intersection area (X∈[-50m,50m], Y∈[0m,100m]), and remove the background point cloud.

[0028] The visual image preprocessing module is involved in: Distortion Correction and Enhancement: Based on Intrinsic Parameter Matrix The image is corrected using distortion coefficients (k1=-0.01, p1=0.002); lighting adaptability is enhanced using the Retinex algorithm (e.g., increasing brightness in backlit scenes, reducing noise in rainy weather), gamma correction γ = 1.5, and dynamic range is compressed to [0, 255]. The corrected RGB image is output (size 1920×1080×3).

[0029] Visual feature alignment: Extract low-level features such as lane lines and vehicle edges from the image (using Canny edge detection) to ensure spatial alignment with the vehicle target area in the radar heatmap (such as the target pixel coordinate range being consistent within the same lane).

[0030] The dynamic weight fusion module includes an environment recognition module for classifying the preprocessed visual image into environmental types and outputting corresponding environmental factors; a dynamic weight calculation module for calculating the LiDAR weight and the visual weight; and a weight superposition fusion module for superimposing the radar heatmap and the visual image according to weights to generate a fusion feature map, and performing normalization and Gaussian smoothing on the fusion feature map to obtain a multimodal fusion feature map. The environment recognition module (lightweight CNN classification network MobileNetV2) classifies visual images into environmental types (sunny / cloudy / rainy / foggy / backlight) and outputs environmental factors (preset values ​​are shown in Table 1).

[0031] Table 1 ; like Figure 2 As shown, the dynamic weight calculation module participates in: Point cloud density confidence score: Count the number of points in a unit area (points / m²). If the density > 100 points / m², the target is clear. If the density is less than 30 points / m² (the target is sparse). , the intermediate value of linear interpolation.

[0032] Reflection intensity confidence: Calculate the variance of the point cloud reflection intensity. Variance < 50 (reflection is stable, such as metal vehicles). Variance > 200 (reflection clutter, such as ground interference). .

[0033] LiDAR point cloud comprehensive confidence level , (range [0,1]).

[0034] Dynamic weight calculation: LiDAR weights and visual weight satisfy Calculation formula: ; in: : Comprehensive confidence level of lidar point cloud (based on point cloud density and reflection intensity variance, range [0,1]); Environmental factors (sunny day 0.3 / cloudy day 0.4 / rainy day 0.8 / foggy day 0.9 / backlight 0.7); Balance coefficient (determined during offline training, with a value of 0.4).

[0035] Example: In rainy weather, Point cloud density confidence, Confidence level of reflection intensity ,but: , 0.83, (LiDAR weighting 83%, vision weighting 17%).

[0036] The weighted summation and fusion module participates in weighted summation and fusion: pixel-level weighted summation and post-processing. The radar heat map and the visual image are superimposed according to weights to generate a fused feature map (384×640×3). Pixel value weighted superposition enter: Radar thermal image: Three-channel pseudo-color image (Pixel value range 0-255); Visual image: Three-channel RGB image (Pixel value range 0-255); Weights: (Radar weights), (Visual weights).

[0037] Fusion formula: Pixel-wise weighted summation to generate fused feature map I fusion : ; In the formula: Represents the fused feature map; Indicates the weight of the lidar; A three-channel pseudo-color image representing a lidar thermal map, with pixel values ​​ranging from 0 to 255; W vis Indicates visual weight, ; The three-channel RGB image represents the visual image, with pixel values ​​ranging from 0 to 255; (u, v) represents the two-dimensional pixel coordinates generated by projecting the preprocessed LiDAR point cloud onto the visual image plane; c represents the RGB channels (0:R, 1:G, 2:B).

[0038] Since the pixel values ​​after weighted stacking may exceed the range of 0-255, linear normalization is used to convert I... fusion Compress to [0,255]: ; Gaussian blur (kernel size=3×3, σ=0.5) is applied to the normalized fused image to reduce noise caused by weight superposition and enhance the stability of subsequent feature extraction.

[0039] The multimodal fusion feature map feature extraction and traffic parameter calculation module includes a MobileNetV3 network model feature extraction module for extracting traffic flow, vehicle speed, queue length and vehicle type classification features from the multimodal fusion feature map; and a traffic flow parameter calculation module for calculating traffic flow, vehicle speed, queue length and vehicle type classification at traffic intersections.

[0040] The MobileNetV3 network model feature extraction module in this embodiment is based on the MobileNetV3-Small framework. Core improvements include a multi-scale feature enhancement module, task decoupling branches, hybrid activation function optimization, and parameter correlation constraints. The specific technical solutions are as follows: (1) Multi-scale feature enhancement module Improved SE attention mechanism: Embed the SE module in each Bottleneck block of MobileNetV3, reducing the compression ratio from 16 to 8 (lowering the channel reduction ratio), enhancing attention to small target features, formula: ; Here, σ is the Sigmoid activation function, and FC is a fully connected layer. By improving the discriminative power of the channel attention weights, the response of key features such as vehicle outline and headlights is enhanced.

[0041] Cross-scale feature pyramid: Fusing feature maps from three scales (1 / 4, 1 / 8, and 1 / 16 of the input resolution), aligning the sizes through upsampling, and then using CBAM (Convolutional Block Attention Module) for spatial-channel attention weighting. ; in F 4. F 8. F 16 The feature maps are at scales of 1 / 4, 1 / 8, and 1 / 16, respectively, to address the problem of feature loss for small targets in traditional lightweight networks.

[0042] (2) Multi-task decoupling branch design To address the specific characteristics of traffic flow parameters, four decoupled output branches are designed to avoid feature conflicts: Traffic flow / speed branch (regression task): It shares the "Global Average Pooling (GAP) + Fully Connected Layer" structure, with the output layer using Linear activation and the loss function being MSE. ; in This represents the actual value (such as traffic flow). These are predicted values.

[0043] Vehicle type classification branch (classification task): Independent convolutional layers (3×3, 128 channels) extract classification-specific features, the output layer uses Softmax activation, and the loss function is cross-entropy. ; in C =3 (number of vehicle types) For true category labels.

[0044] Queue length branch (structured regression task): A coordinate attention module is introduced to enhance the spatial location sensitivity of the vehicle cluster. The output layer uses L1 loss. ; in This represents the actual queue length at the entrance lane.

[0045] Multi-task loss fusion: The weights are dynamically adjusted based on task difficulty, using the following formula: ; Among them, weight w The validation set accuracy is adaptively updated (e.g., the weight of queue length is increased on rainy days).

[0046] (3) Lightweight optimization strategy Hybrid activation functions: L-ReLU6 is used in the shallow layer (feature extraction layer) to solve the vanishing ReLU gradient problem, and HardSwish is used in the deep layer (task branch) to improve feature representation, resulting in a 15% improvement in computational efficiency; Dynamic channel pruning: Redundant channels are pruned based on feature importance (L1 norm), while core feature channels (such as vehicle contour channels) are retained, reducing the number of parameters by 30%. INT8 quantization: Weights and activation values ​​are quantized from FP32 to INT8, compressing the model size by 60% and improving edge inference speed by 2 times.

[0047] (4) Parameter association constraints By leveraging the physical correlations between traffic flow parameters (e.g., increased traffic volume → increased queue length), a correlation loss function is designed: ; in Q Queue length V For traffic flow, Corr is the Pearson correlation coefficient, which forces the model to learn the intrinsic relationship between parameters and improves robustness in complex scenarios.

[0048] The traffic flow parameter calculation module participates in the calculation of traffic flow parameters: Traffic flow: Based on the detection of bounding boxes using YOLOv5s lightweight version, combined with SORT algorithm trajectory tracking, the traffic flow is counted by entrance lane (vehicles / 5 minutes). Vehicle speed: Calculated by the displacement of the target center pixel in adjacent frames, formula: Where Z is the target distance (measured by radar). f x For camera focal length, fps For frame rate; Queue length: DBSCAN clusters the point cloud vehicle cluster and calculates the distance between the first and last points (m); Vehicle type classification: Based on point cloud size (length / width / height) and visual features (height-to-width ratio), vehicles are classified into small vehicles (length < 4.5m), medium-sized vehicles (length 4.5-6m), and large vehicles (length > 6m).

[0049] Taking "traffic flow detection at urban four-way intersections" as an example, the implementation steps are explained in detail: Step 1: Equipment Deployment and Calibration The lidar was installed on a pole at the intersection (6m high, 15° downward angle), and the cameras were coaxially mounted (0.5m spacing); extrinsic parameter calibration: a checkerboard pattern was placed at distances of 5m / 10m / 20m, and 20 sets of samples were collected. The results were: .

[0050] Step 2: Data Acquisition and Preprocessing The lidar outputs a point cloud frame (40,000 points) every 100ms, and the camera outputs an image frame (1920×1080) every 40ms. Rainy day preprocessing: radar point cloud denoising (removing points with reflection intensity <200), image enhancement (Retinex algorithm γ=1.5).

[0051] Step 3: Dynamic Weight Fusion The environmental identification submodule outputs "Rainy Day" (confidence level 0.92), environmental factors. ; Radar point cloud confidence (Density 15 points / m²), calculate the weights. 0.78.

[0052] Step 4: Parameter Output Lightweight CNN inference output: Traffic flow: 12 vehicles / 5 minutes at the east entrance, 8 vehicles / 5 minutes at the south entrance; Vehicle speed: Average 25km / h at the east entrance (maximum 32km / h); Queue length: 35m (5 vehicles) for the south entrance straight lane; Vehicle types: 15 small cars, 3 medium cars, and 2 large cars.

[0053] Finally, the above system was tested on an edge device (Jetson Nano 4GB), and the system performance is as follows: Inference latency: Single frame processing time ≤ 80ms (including feature extraction + 4-branch output), meeting the 25fps real-time requirement; Model size: 8.5MB (after quantization), supports local storage and updates of roadside units; Detection accuracy: Traffic volume MAE ≤ 5 vehicles / hour (40% reduction compared to traditional visual methods); Vehicle speed RMSE ≤ 3km / h (and ≤ 4km / h in rainy conditions); Queue length error ≤3m (for large vehicle queues); Vehicle type classification accuracy is ≥92% (small cars ≥95%, large cars ≥88%).

[0054] Through the implementation of the above embodiments, high-precision, real-time acquisition of traffic flow parameters in complex environments has been achieved, providing core data support for intelligent traffic signal control and road network optimization.

[0055] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A method for detecting traffic parameters in complex environments based on the fusion of lidar and vision, characterized in that, Includes the following steps: S1. Install a LiDAR and a camera, calibrated using the checkerboard joint calibration method, coaxially above the traffic intersection, and simultaneously acquire the LiDAR point cloud data collected by the LiDAR and the visual image data collected by the camera. S2. Preprocess the lidar point cloud data and visual image data respectively; S3. Project the preprocessed lidar point cloud onto the visual image plane to generate two-dimensional pixel coordinates, and generate a radar heat map by encoding the reflection intensity and density. The 3D point cloud is converted into a 2D feature map aligned with the visual image space to highlight the vehicle target area; the point cloud density and reflection intensity variance of the lidar are calculated, and the lidar point cloud density confidence and reflection intensity confidence are obtained after normalization of the lidar point cloud density and reflection intensity confidence. The lidar point cloud comprehensive confidence is obtained by combining the lidar point cloud density confidence and reflection intensity confidence. S4. Classify the preprocessed visual image by environment type and environment recognition factor according to the preset environment type and output the corresponding environment factor; S5. Combine the comprehensive confidence level of the lidar point cloud with environmental factors to obtain the lidar weight, and obtain the visual weight based on the characteristic that the sum of the lidar weight and the visual weight is equal to 1. S6. Superimpose the radar heat map and the visual image according to the weights to generate a fused feature map, and perform normalization and Gaussian smoothing on the fused feature map to obtain a multimodal fused feature map. S7. Input the multimodal fusion feature map into the improved MobileNetV3 network model for feature extraction. The features include traffic flow, vehicle speed, queue length and vehicle type classification features. Calculate the traffic flow, vehicle speed, queue length and vehicle type classification of the traffic intersection based on the extracted traffic flow, vehicle speed, queue length and vehicle type classification features.

2. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 1, characterized in that, In step S1, the GPS timing module outputs a PPS second pulse signal to synchronously trigger the lidar and camera to collect lidar point cloud data and visual image data, with a timestamp accuracy of ≤1μs.

3. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 1, characterized in that, In step S2, the preprocessing of the LiDAR point cloud data includes denoising, downsampling, and ROI cropping; the preprocessing of the visual image data includes distortion correction and enhancement and visual feature alignment. Visual feature alignment is achieved by extracting lane lines and vehicle edge features from the visual image through Canny edge detection to ensure that the visual image is spatially aligned with the vehicle target area in the LiDAR heatmap.

4. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 1, characterized in that, In step S3, the formula for calculating the overall confidence level of the lidar point cloud is: ； In the formula: This represents the overall confidence level of the lidar point cloud, ranging from [0,1]. This indicates the confidence level of the lidar point cloud density. If the lidar point cloud density is >100 points / m², then... 2 , If the point cloud density of the lidar is less than 30 points / m 2 , ; This represents the confidence level of the lidar point cloud reflection intensity. If the variance of the lidar point cloud reflection intensity is less than 50, If the variance of the reflectance intensity of the lidar point cloud is > 200, .

5. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 1, characterized in that, In step S4, the preset environment types and environment identification factors are shown in the table below: 。 6. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 5, characterized in that, In step S5, the formula for calculating the lidar weight is: ； In the formula: Indicates the weight of the lidar; This represents the overall confidence level of the lidar point cloud, ranging from [0,1]. Indicates environmental factors; This represents the balance coefficient, with a value of 0.

4.

7. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 6, characterized in that, In step S6, the formula for generating the fused feature map is: ； In the formula: Represents the fused feature map; Indicates the weight of the lidar; A three-channel pseudo-color image representing a lidar heat map, with pixel values ​​ranging from 0 to 255; Indicates visual weight, ; A three-channel RGB image representing a visual image, with pixel values ​​ranging from 0 to 255; This represents the two-dimensional pixel coordinates generated by projecting the preprocessed LiDAR point cloud onto the visual image plane. This represents the RGB channels (0:R, 1:G, 2:B).

8. The method for detecting traffic parameters in complex environments based on the fusion of lidar and vision according to claim 1, characterized in that, In step S7: Traffic flow: Based on the detection of target boxes using YOLOv5s lightweight version, combined with SORT algorithm trajectory tracking, the count is based on the entrance lane, vehicles / 5 minutes; Vehicle speed: Calculated by the displacement of the target center pixel in adjacent frames, formula: ;in Z The distance to the target vehicle measured by lidar. f x For camera focal length, fps For frame rate; Queue length: DBSCAN clusters point cloud vehicle clusters and calculates the distance between the first and last points, in meters; Vehicle type classification: Based on the length / width / height of the LiDAR point cloud and the aspect ratio of the visual image, vehicles are classified into small vehicles (length < 4.5m), medium-sized vehicles (length 4.5-6m), and large vehicles (length > 6m).

9. A traffic parameter detection system for complex environments based on the fusion of lidar and vision, characterized in that, include: The multimodal data acquisition and preprocessing module includes a LiDAR and a camera coaxially mounted above a traffic intersection and calibrated using a checkerboard joint calibration method; a GPS timing module for outputting PPS (pixel per second) signals to synchronously trigger the LiDAR and camera to acquire LiDAR point cloud data and visual image data; a LiDAR point cloud data preprocessing module for denoising, downsampling, and ROI cropping of the LiDAR point cloud data; and a visual image data preprocessing module for distortion correction, enhancement, and visual feature alignment of the visual image data. The dynamic weight fusion module includes an environment recognition module for classifying the preprocessed visual image into environmental types and outputting corresponding environmental factors. And a dynamic weight calculation module for calculating lidar weights and visual weights; and a weight superposition fusion module for superimposing the lidar heatmap and visual image according to weights to generate a fusion feature map, and performing normalization and Gaussian smoothing on the fusion feature map to obtain a multimodal fusion feature map. The multimodal fusion feature map feature extraction and traffic parameter calculation module includes a MobileNetV3 network model feature extraction module for extracting traffic flow, vehicle speed, queue length and vehicle type classification features from the multimodal fusion feature map; And a traffic flow parameter calculation module for calculating traffic volume, vehicle speed, queue length, and vehicle type classification at traffic intersections.

Images