A multi-scale target detection method and system for road scenes

Abstract

The present invention relates to a method and system for multi-scale target detection in road scenes. By integrating complementary data from visual cameras, lidar, and infrared sensors, combined with hardware acceleration and dynamic algorithm optimization, it achieves high-precision, real-time detection of targets in all scenarios and at multiple scales. Using a collaborative architecture of FPGA and GPU, preprocessing tasks such as image denoising, point cloud filtering, and heat map enhancement are completed at the edge, and a unified input is generated through multimodal feature fusion. Based on the dynamic clustering and non-maximum suppression (Soft-NMS) optimization mechanism of the Gaussian mixture model (GMM), the system adaptively distinguishes target scales and adjusts the suppression strategy, significantly reducing the missed detection rate of small targets. At the same time, the cloud-edge collaborative scheduling strategy supports local millisecond-level response to emergencies and global parameter optimization in the cloud, balancing real-time performance and computational depth. This solution can be widely used in the fields of intelligent traffic management, monitoring, and autonomous driving, providing efficient and reliable technical support for target detection and early warning in complex road scenes.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, intelligent transportation, and public safety technologies. Specifically, it relates to a multimodal data fusion framework based on visual cameras, lidar, and infrared sensors. This framework combines a Gaussian mixture model (GMM) with an adaptive dynamic optimization algorithm (Soft-NMS parameter dynamic adjustment) to achieve high-precision detection and real-time warning of multi-scale targets in complex road scenarios. Background Art

[0002] In the fields of intelligent transportation and public safety, object detection technology is the core foundation for real-time monitoring and emergency response. However, existing technical solutions face multiple challenges in complex road scenarios.

[0003] Detection systems based on a single sensor experience significant performance degradation in extreme environments such as low light, rain, fog, and strong backlight, making it difficult to stably capture target features. This is especially true at night or in inclement weather, where the missed detection rate for small targets remains high. Furthermore, when processing dense or occluded scenes, traditional target detection algorithms often mistakenly delete overlapping candidate boxes due to the fixed-threshold non-maximum suppression (NMS) strategy, resulting in the loss of key targets. At the hardware level, centralized processing architectures are limited by data transmission bandwidth and computing resource allocation, making it difficult to meet the millisecond-level real-time response requirements of large-scale monitoring scenarios. This is especially true during peak traffic or emergencies, where system delays can pose safety risks.

[0004] The 2024 paper "Radar-Camera 3D Object Detection Based on Mutual Guidance and Complementation of Multimodal Information" proposed a radar-camera 3D object detection model based on mutual guidance and complementation of multimodal information. This model adaptively selects information from another modality for feature fusion. This adaptive information fusion involves mutual guidance and complementation at both the data and feature levels. While this approach addresses the issue of network performance degradation caused by the fusion of irrelevant information, performance degradation still exists in certain complex scenarios.

[0005] In 2024, the paper "Target Detection and Tracking Using Highway-Side Radar and Camera Fusion" proposed a multimodal fusion target detection and tracking method based on highway-side millimeter-wave radar (MWR) and cameras. This method addresses the issue of reduced detection and positioning accuracy caused by target occlusion and external environmental interference in the development of ITS. However, RCFTrack is prone to tracking interruptions during long tracking cycles.

[0006] In 2023, the paper "Improved YOLOv3 model with feature map cropping for multi-scale road object detection" proposed a method based on YOLOv3 (You Only Look Once v3). The K-means-GIoU algorithm was designed to generate a priori bounding boxes with shapes close to the ground-truth boxes, significantly reducing training complexity. A detection branch was then added to detect small objects. A feature map cropping module was introduced to this newly added detection branch, padding the feature maps of the cropped areas with zeros. Furthermore, channel and spatial attention modules were added to enhance the network's focus on key areas. However, this algorithm lacked optimizations to improve detection efficiency.

[0007] The 2023 paper "BL-YOLOv8: An Improved Road Defect Detection Model Based on YOLOv8" proposed an enhanced road defect detection algorithm, BL-YOLOv8, based on the YOLOv8s model. By incorporating the BiFPN concept, the algorithm reconstructed the YOLOv8s model's core structure, optimizing the model and reducing the number of parameters, computational burden, and overall size. However, this algorithm is only suitable for specific types of road images, and its adaptability to different cities and road types requires further verification and optimization.

[0008] In 2021, the paper "An Improved Swin Transformer-Based Model for Remote Sensing Object Detection and Instance Segmentation" proposed an improved Swin transformer based on the advantages of transformers and CNNs. The paper also designed a local perception Swin transformer (LPSW) backbone network to enhance the network's local perception capabilities and improve the detection accuracy of small-scale objects. However, its adaptability and generalization capabilities for multi-scenario road detection have not yet been verified.

[0009] While current improvements attempt to enhance detection accuracy through multi-scale feature fusion or algorithm optimization, they still have significant limitations. For example, single-modality data cannot achieve cross-environmental robustness, and traditional hardware acceleration solutions lack dynamic task scheduling capabilities, making it difficult to balance real-time performance with computational depth. Therefore, a technical solution is urgently needed that can fuse multi-source sensor data, adaptively optimize detection algorithms, and achieve efficient processing through hardware-algorithm collaborative acceleration.

[0010] Existing technologies face significant limitations in object detection in complex scenes. First, traditional systems overly rely on a single sensor (such as a visual camera), resulting in severe environmental adaptability issues. Second, existing algorithms perform poorly when handling small objects. This is primarily due to the traditional non-maximum suppression (NMS) strategy, which uses a fixed threshold to suppress overlapping boxes, resulting in the inadvertent deletion of small objects in dense scenes. Third, the inference latency of traditional deep learning models (such as Faster R-CNN) on CPUs exceeds 100 milliseconds per frame (at 1080p resolution), making it difficult to meet millisecond-level response requirements. Finally, insufficient robustness in dynamic scenes limits the system's practicality. In scenes with high-speed movement or complex occlusion, object detection boxes are prone to jitter or missed detections. These issues collectively highlight the imbalance between environmental adaptability, efficiency, and cost in existing technologies, providing clear technical improvement directions for innovative solutions. Summary of the Invention

[0011] The objectives of the present invention are achieved through the following technical solutions.

[0012] Based on this, the present invention integrates vision, lidar, and infrared sensors to construct a multimodal data fusion framework. This framework, combined with dynamic clustering and non-maximum suppression (Soft-NMS) optimization using a Gaussian mixture model (GMM), overcomes the limitations of single-sensor perception. This invention utilizes a collaborative acceleration architecture combining FPGA edge preprocessing with GPU parallel inference, combined with cloud-edge dynamic task scheduling, to significantly improve the detection accuracy and real-time response capabilities of multi-scale targets in complex road scenarios. This solution performs exceptionally well in scenarios such as nighttime monitoring and traffic diversion in rain and fog, providing reliable technical support for intelligent traffic management and public safety.

[0013] The main steps of the present invention are as follows:

[0014] Step 1: Multi-sensor coordinated deployment and spatiotemporal calibration.

[0015] Visual cameras, lidar, and infrared sensors are deployed in the monitoring area to ensure overlapping fields of view and no physical obstructions. Sensor spatial calibration and time synchronization are achieved through a checkerboard calibration board and hardware synchronization signals.

[0016] Step 2: FPGA edge preprocessing and multimodal data fusion.

[0017] Raw data is fed into the FPGA for parallel pipeline processing: visual images undergo median filtering for denoising (2ms), LiDAR point clouds undergo voxel filtering for downsampling (1.5ms), and infrared thermal images undergo contrast enhancement using CLAHE (Complementary Electron Heterogeneous Heterogeneity) (1ms). After preprocessing, the data is aligned in real time, and the point cloud is projected onto the image plane to generate a depth confidence map. This map is then concatenated with the RGB image and thermal map channel by channel to form a five-channel fused feature map (H×W×5), providing a unified input for subsequent detection.

[0018] Step 3: GPU parallel inference and dynamic optimization.

[0019] The fused feature map is fed into the GPU, where Faster RCNN generates candidate boxes and extracts geometric and modal features (width, height, depth confidence, and thermal radiation intensity). Based on GMM clustering, the candidate boxes are classified into three categories: large, medium, and small. The σ value of Soft-NMS is dynamically adjusted (σ = 0.2 for small objects and σ = 0.6 for large objects). A weighted score is then applied based on depth and thermal radiation data to select high-confidence detection boxes, ultimately outputting the object's location and category.

[0020] Step 4: Cloud-edge collaborative response and global optimization.

[0021] Edge devices trigger alarms in real time (latency <50ms), prioritizing local response for emergencies (such as explosive detection). Non-emergency data is uploaded to the cloud, and historical data from multiple nodes is aggregated to train incremental models. GMM clustering centers and σ value strategies are optimized, and updated parameters are fed back to edge devices. Dynamic scheduling balances real-time performance with computational depth, ensuring long-term system stability and accuracy.

[0022] The present invention has the following beneficial effects:

[0023] This invention demonstrates significant technical advantages in complex road scenarios through multimodal sensor fusion and dynamic optimization algorithms. First, the system integrates complementary data from visual, lidar, and infrared sensors, overcoming the limitations of traditional single-sensor perception in extreme environments. To address the challenges of multi-scale target detection, this invention proposes an optimization mechanism based on dynamic clustering. This dynamic optimization capability enables the system to achieve higher overall accuracy in complex scenarios (such as rush hour or nighttime monitoring), particularly improving the recognition of static or partially occluded targets compared to traditional algorithms. Regarding real-time performance, the combination of a hardware collaborative acceleration architecture and a cloud-edge collaborative scheduling strategy significantly enhances system response efficiency. Furthermore, by dynamically allocating computing tasks, the system optimizes resource utilization while ensuring real-time performance, reducing hardware deployment and maintenance costs. Furthermore, the system's environmental robustness is significantly enhanced. The fusion and correction technology for multi-sensor data effectively suppresses motion blur and occlusion interference, improving detection stability for high-speed moving targets or in complex backgrounds. This feature makes it widely applicable in fields such as intelligent traffic management, public safety monitoring, and autonomous driving, adapting to the diverse needs of changing scenarios and providing reliable technical support for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0025] Figure 1 FIG. 4 shows a multimodal sensor fusion architecture diagram according to an embodiment of the present invention.

[0026] Figure 2 A flow chart of a dynamic Soft-NMS optimization algorithm according to an embodiment of the present invention is shown.

[0027] Figure 3 A schematic diagram of FPGA-GPU hardware acceleration according to an embodiment of the present invention is shown.

[0028] Figure 4 A diagram of a cloud-edge collaboration and dynamic scheduling framework according to an embodiment of the present invention is shown.

[0029] Figure 5 A comprehensive performance verification diagram according to an embodiment of the present invention is shown.

[0030] Figure 6 A schematic diagram of the infrared and visible light fusion effect according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] The present invention relates to a method and system for multi-scale target detection in road scenes. By integrating complementary data from visual cameras, lidar, and infrared sensors, combined with hardware acceleration and dynamic algorithm optimization, it achieves high-precision, real-time detection of targets at multiple scales across all scenarios. The system utilizes a collaborative architecture of FPGAs and GPUs to complete preprocessing tasks such as image denoising, point cloud filtering, and heatmap enhancement at the edge, generating a unified input through multimodal feature fusion. Dynamic clustering and non-maximum suppression (Soft-NMS) optimization mechanisms based on Gaussian mixture models (GMMs) adaptively distinguish target scales and adjust suppression strategies, significantly reducing the missed detection rate of small targets. Furthermore, a cloud-edge collaborative scheduling strategy supports local millisecond-level response to emergencies and global parameter optimization in the cloud, balancing real-time performance with computational depth.

[0033] like Figure 1 As shown, the multi-scale target detection method for road scenes of the present invention is based on multimodal sensor fusion and specifically includes the following steps:

[0034] Step 1: Multi-source data collection and spatiotemporal alignment. This includes sensor deployment and initialization, as well as raw data collection.

[0035] Step 2: Edge preprocessing and feature extraction. This includes FPGA real-time preprocessing and multimodal feature fusion.

[0036] Step 3: Multi-scale object detection and dynamic optimization. This includes candidate box generation and feature extraction, Gaussian mixture model (GMM) clustering, and dynamic Soft-NMS optimization.

[0037] Step 4: Hardware acceleration and cloud-edge collaborative response. This includes FPGA-GPU collaborative acceleration, real-time alerting, and cloud-side feedback.

[0038] Preferably, in step 1, the sensor configuration is specifically as follows:

[0039] a. Visual camera: 2 megapixels, 30fps, responsible for capturing RGB images and motion information. Deployed at a high point in the monitoring area, it covers the target detection range and ensures an unobstructed view.

[0040] b. LiDAR: 16 beams, 100m detection range, outputs point cloud data (XYZ coordinates) and depth confidence. Deployed at a high point in the monitoring area, covering the target detection range and ensuring an unobstructed view.

[0041] c. Infrared camera: Thermal sensitivity ≤ 50mK, generating a thermal radiation intensity map (unit: W / m²). Mounted side by side with the visual camera, with a field of view overlap of ≥ 90%.

[0042] Preferably, in step 1, the calibration and synchronization parts can be divided into:

[0043] a. Spatial alignment: Align the LiDAR coordinate system with the visual camera coordinate system using a calibration matrix, with a calibration error of <0.1 pixel.

[0044] b. Time synchronization: Use hardware trigger signal (PPS pulse) to achieve multi-sensor frame synchronization (timing error <5ms).

[0045] Different sensors collect different data. The types of collected data are as follows:

[0046] a. The visual camera outputs an RGB image sequence;

[0047] b. LiDAR generates point cloud data in real time (updated at 10Hz);

[0048] c. Infrared camera outputs thermal radiation intensity map (30fps).

[0049] Preferably, in step 2, FPGA real-time preprocessing is divided into the following three steps:

[0050] a. Image denoising: Use a median filter (kernel size 3×3) to remove impulse noise from the visual image. The denoised RGB image maintains a resolution of 1920×1080.

[0051] b. Point cloud downsampling: Use voxel filtering (voxel size 0.1m³) to compress the LiDAR point cloud data. The downsampled point cloud data retains key obstacle information.

[0052] c. Thermal Image Enhancement: This algorithm uses the CLAHE algorithm (Contrast Limited Adaptive Histogram Equalization) to enhance infrared image details. The enhanced thermal image features improved detail clarity.

[0053] Preferably, in step 2, the multimodal feature fusion includes:

[0054] Feature alignment: Project the lidar point cloud onto the visual image plane to generate a depth confidence map (resolution 1920×1080);

[0055] Multimodal feature map: The RGB image, depth confidence map, and thermal radiation intensity map are concatenated channel by channel to form a fused feature map (size: H×W×5). The fusion of infrared and visible light effectively preserves texture details and enhances target representation capabilities.

[0056] Preferably, in step 3, during candidate box generation and feature extraction:

[0057] Object detection model: Based on the Faster R-CNN framework, it inputs a fused feature map and outputs an initial candidate box.

[0058] Feature vector construction: Extract the following features for each candidate box:

[0059] Geometric features: candidate box width (w), height (h), and area (w×h);

[0060] Modal features: depth confidence (dconf), thermal radiation intensity (tintensity).

[0061] Preferably, in step 3, the Gaussian mixture model (GMM) clustering is as follows:

[0062] Input data: feature vector F=[w,h,dconf,tintensity] of all candidate boxes;

[0063] EM algorithm modeling:

[0064] Initialization: Randomly set the mean (μ1, μ2, μ3) and covariance matrix (Σ1, Σ2, Σ3) of three Gaussian distributions;

[0065] Iterative optimization: E-step (Expection-Step): Calculate the posterior probability of each candidate box belonging to each Gaussian distribution; M-step (Maximization-Step): Update the Gaussian distribution parameters and maximize the log-likelihood function logL;

[0066] Convergence condition: Change in log-likelihood Or the number of iterations ≥ 100; where logL represents the log-likelihood function, which is used to measure the degree of fit of the GMM model to the data. It means that the change of the log-likelihood function is very small and the model has converged. The number of iterations ≥ 100 is to prevent the algorithm from falling into an infinite loop.

[0067] Output: 3 types of Gaussian distribution labels (large target, medium target, small target) and distribution parameters (μ, Σ).

[0068] Preferably, in step 3, Figure 2 As shown, the dynamic Soft-NMS optimization algorithm process is as follows:

[0069] Candidate box sorting: sort by initial confidence (Score) from high to low;

[0070] The dynamic suppression strategy dynamically adjusts the confidence of the candidate box through the attenuation function, and adaptively suppresses redundant boxes according to the target size and overlap degree:

[0071] The decay function is:

[0072]

[0073] Where (k=1, 2, 3 corresponds to small / medium / large targets); i and j are the numbers of the candidate boxes, used to distinguish different boxes; Indicates the candidate box currently being processed. Represents another candidate box to compare with.

[0074] σ k Values: small target (σ1=0.2), medium target (σ2=0.4), large target (σ3=0.6);

[0075] Multimodal weighting: Fusion of depth confidence and thermal radiation intensity into the final score:

[0076]

[0077] d conf : depth confidence of the lidar (0~1);

[0078] t intensity : thermal radiation intensity of the infrared camera (normalized to 0~1);

[0079] Screening threshold: Keep Score final Candidate boxes with a value of ≥0.5.

[0080] Preferably, in step 4, Figure 3 As shown, the FPGA-GPU collaborative acceleration is as follows:

[0081] FPGA pipeline (total latency ≤ 5ms): Image denoising (2ms) → Point cloud filtering (1.5ms) → Thermal image enhancement (1ms);

[0082] GPU parallel computing (total latency ≤ 25ms):

[0083] Faster R-CNN inference (10ms) → GMM clustering (8ms) → Soft-NMS optimization (7ms).

[0084] Preferably, in step 4, Figure 4 As shown, the real-time alarm and cloud feedback are as follows:

[0085] Edge response: Detection results are output via NVIDIA Jetson AGX Xavier, triggering audible and visual alarms (latency < 50ms); local response is prioritized for emergencies.

[0086] Cloud optimization: Aggregate multi-node data, train incremental models, and update GMM cluster centers; dynamically adjust FPGA preprocessing parameters (such as filter kernel size).

[0087] It has been verified that the system of the present invention achieves an mAP of 93.8% by fusing RGB images, lidar depth maps and infrared thermal maps, combined with a dynamic optimization algorithm, which is 7.6% higher than the single vision system (86.2%).

[0088] For small target detection, this paper adopts a lightweight network and dynamic SoftNMS (σ=0.2), and the recall rate is increased to 92.3%, which is 13.8% higher than YOLOv3 (78.5%).

[0089] like Figure 5 As shown in the figure, in large object detection, the recall rate is slightly increased to 93.1% (baseline 91.4%) and the missed detection rate is reduced to 6.1% (traditional NMS is 12.4%) through the ResNet50 network and dynamic SoftNMS (σ=0.6).

[0090] Through the FPGAGPU collaborative acceleration architecture (FPGA preprocessing delay ≤ 5ms, GPU inference delay ≤ 25ms), the full-process delay is reduced from > 100ms of pure CPU to 29.5ms, a reduction of 70.5ms.

[0091] The dynamic SoftNMS strategy combined with multimodal weighted scoring effectively reduces the accidental deletion of small targets and reduces the missed detection rate by 6.3%.

[0092] The cloud aggregates historical data from multiple nodes and dynamically updates the GMM cluster center and SoftNMS parameters. After optimization, the system mAP reaches 92.9% (the original system was 88.2%), an increase of 4.7%.

[0093] Figure 6 A schematic diagram of the infrared and visible light fusion effect according to an embodiment of the present invention is shown. This solution can be widely used in intelligent traffic management, monitoring, and autonomous driving, providing efficient and reliable technical support for target detection and early warning in complex road scenarios.

[0094] The foregoing description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-scale target detection method for road scenes based on multimodal sensor fusion, characterized by: The steps include: Step 1: Synchronously collect raw data of the road scene through visual cameras, lidar, and infrared sensors, and complete sensor spatial calibration and time synchronization through a checkerboard calibration board and hardware synchronization signals; Step 2: De-noise the visual image, downsample the lidar point cloud, and enhance the infrared thermal radiation intensity map; Step 3: Fuse the processed multimodal data into a fused feature map; Step 4: Extract candidate boxes from the fused feature map and extract geometric and modal features. Cluster the candidate boxes using the Gaussian mixture model (GMM) and classify them into three categories: large, medium, and small according to the target size. Step 5: Dynamic Soft-NMS optimization, dynamically adjust detection parameters according to target categories, and select high-confidence candidate boxes; Step 6: Output the final target location and category, triggering real-time alarms and cloud feedback; The step 4 includes: Based on the Faster R-CNN framework, the fused feature map is input and the initial candidate box is output; Feature vector construction: Extract the following features for each candidate frame: Geometric features: candidate frame width, height, and area; Modal features: depth confidence and thermal radiation intensity; Gaussian mixture model GMM clustering is as follows: Input the feature vectors of all candidate boxes; EM algorithm modeling: Initialization: Randomly set the mean and covariance matrix of three Gaussian distributions; Iterative optimization: E step: calculate the posterior probability of each candidate box belonging to each Gaussian distribution; M step: update the Gaussian distribution parameters and maximize the log-likelihood function; Convergence condition: Log-likelihood change |ΔlogL|<1e −5 Or the number of iterations ≥ 100; where logL represents the log-likelihood function; Output: 3 types of Gaussian distribution labels and distribution parameters.

2. The method according to claim 1, wherein In step 1: The sensor spatial calibration and time synchronization are completed by using the checkerboard calibration board and the hardware synchronization signal, including: Spatial alignment: Align the LiDAR coordinate system with the visual camera coordinate system through the calibration matrix; Time synchronization: Use hardware trigger signal PPS pulse to achieve multi-sensor frame synchronization.

3. The method according to claim 1 or 2, wherein: The visual camera outputs an RGB image sequence, the laser radar generates point cloud data in real time, and the infrared sensor outputs an infrared thermal radiation intensity map.

4. The method according to claim 3, wherein The step 2 includes: The RGB image sequence is subjected to 3×3 median filtering; the lidar point cloud is subjected to voxel filtering; and the infrared thermal radiation intensity image is subjected to contrast enhancement using the CLAHE algorithm.

5. The method according to claim 4, wherein In step 3, the multimodal data is fused into a fusion feature map, including: Feature alignment: Project the lidar point cloud onto the visual image plane to generate a depth confidence map; The RGB image, depth confidence map, and infrared thermal radiation intensity map are spliced by channel to form a fusion feature map.

6. The method according to claim 1 or 5, wherein: In step 5, the dynamic Soft-NMS optimization algorithm process is as follows: Candidate box sorting: sort by initial confidence from high to low; The dynamic suppression strategy dynamically adjusts the confidence of the candidate box through the attenuation function, and adaptively suppresses redundant boxes according to the target size and overlap degree; Multimodal weighting: Fusion of depth confidence and thermal radiation intensity into the final score.

7. The method according to claim 6, wherein Step 5 includes: using a Soft-NMS parameter of σ=0.2 for small targets and σ=0.6 for large targets; final score = 0.6×depth confidence + 0.4×thermal radiation intensity; retaining candidate boxes with a score ≥0.

5.

8. The method according to claim 1, wherein In step 6, large targets are processed by the ResNet-50 network, and small targets are processed by the lightweight network; the edge trigger alarm delay is less than 50ms, and the cloud optimizes model parameters daily.

9. The method according to claim 1, wherein The cloud feedback includes: aggregating multi-node historical data to train incremental models, optimizing GMM clustering centers and σ value strategies, and updating parameters to feed back to edge devices.

Images