Vision-based haze unmanned aerial vehicle aerial photography vehicle real-time detection method and device

Abstract

The invention discloses a real-time detection method and device for an aerial vehicle of a haze unmanned aerial vehicle based on vision, and the method comprises the steps: designing a new defogging module FMGRModel through the fusion of multi-scale fog concentration sensing and physical drive residual learning for the influence of a haze environment; in order to solve the problem of low vehicle detection precision, a new structure PrFmFPN is provided, higher-level feature fusion is realized, and a CA attention module and a DG-CAFM enhancement module are added; aiming at real-time detection requirements of an airborne platform, a new S-FocalerIoU loss function is fused to accelerate matching and convergence; inputting a training set of a vehicle image data set into the improved model for learning so as to obtain a trained network model MgFm-YOLO; and inputting to-be-detected target vehicle image data in the haze weather into the MgFm-YOLO model, and detecting position information of the target vehicle. On the premise of considering the accuracy and the reasoning speed, the system can adapt to complex environments such as haze weather and is embedded into an airborne hardware platform of the unmanned aerial vehicle to detect the vehicle in real time.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a vision-based method and apparatus for real-time detection of drone aerial vehicles in hazy weather. Background Technology

[0002] In recent years, drones have received widespread attention in the field of target detection due to their advantages such as maneuverability, flexibility, and wide field of view. They can collect data in real time from different locations, angles, and altitudes, and can even reach dangerous and inaccessible areas. Simultaneously, with rapid urbanization and a surge in vehicle ownership, smog (air pollution caused by particulate matter such as PM2.5 and PM10, as well as aerosols) is becoming increasingly frequent, posing serious challenges to traffic management, public safety, and environmental monitoring. Therefore, in this context, it is particularly important to develop a method and device suitable for real-time vehicle detection in smoggy conditions, capable of embedding drone hardware. This technology can be applied not only to emergency command, pollution management, and smart city scenarios, but also has significant practical value and social significance.

[0003] Compared to other detection methods and devices, real-time vehicle detection using drones in hazy weather presents more difficulties and challenges. First, drones typically operate at altitudes of 50-200 meters, resulting in a predominance of small-sized vehicle targets in the collected visual data, significantly increasing the difficulty of target detection. Second, in hazy weather, the background of roads, buildings, and vegetation appears grayish-white, further blurring edges and textures and increasing the difficulty of vehicle detection. Third, the shaking of the drone during hovering and the fact that vehicles are usually in motion during image acquisition lead to spatial motion blur in the data. Fourth, drones typically require high real-time inference speeds to meet data processing needs while supporting the simultaneous operation of key functions such as automatic obstacle avoidance and dynamic path planning, placing stringent demands on the lightweight nature and computational efficiency of the algorithm. Fifth, the limited hardware resources of the drone's onboard platform make it difficult to deploy computationally intensive large-scale deep learning models, severely restricting the widespread application of high-performance detection algorithms in engineering practice. Summary of the Invention

[0004] To address the problems mentioned above, this application provides a lightweight detection method and apparatus based on vision that can adapt to complex environments such as haze and be embedded in an UAV onboard hardware platform for real-time vehicle detection. This aims to solve the technical problem in the prior art of effectively detecting vehicle targets in visual data captured by UAV aerial photography in real-time under complex environments such as haze.

[0005] Firstly, this application provides a vision-based method for real-time vehicle detection in haze drone aerial photography, specifically including:

[0006] Step 1: Obtain a sample image dataset taken by a drone under hazy weather, and extract the vehicle image dataset from the sample image dataset;

[0007] Step 2: Preprocess the vehicle image dataset and use the preprocessed vehicle image dataset as the training set; the training set includes: target vehicle image data and corresponding labels, the labels being used to indicate the location information of the target vehicle;

[0008] Step 3: Train the pre-established improved YOLO target detection model based on the training set to obtain a vehicle target detection model, wherein the improved YOLO target detection model is the MgFm-YOLO model;

[0009] Step 4: Input the target vehicle image data under the hazy weather into the vehicle target detection model to detect the location information of the target vehicle.

[0010] Secondly, this application provides a vision-based real-time vehicle detection device for drone aerial photography in haze, specifically including:

[0011] Data acquisition module: used to acquire sample image datasets taken by drones under hazy weather, and extract vehicle image datasets from the sample image datasets;

[0012] The data preprocessing module is used to preprocess the vehicle image dataset and use the preprocessed vehicle image dataset as a training set; the training set includes: target vehicle image data and corresponding labels, the labels being used to indicate the location information of the target vehicle;

[0013] The training module is used to train a pre-established improved YOLO object detection model based on the training set to obtain a vehicle object detection model, wherein the improved YOLO object detection model is the MgFm-YOLO model.

[0014] The detection module is used to input the image data of the target vehicle under hazy weather into the vehicle target detection model to detect the location information of the target vehicle.

[0015] This application provides a vision-based method and device for real-time vehicle detection by drones in hazy weather. It integrates with the drone's onboard hardware platform to monitor vehicles in real time, enabling real-time monitoring in complex environments such as haze. This method is beneficial for applications in emergency command, pollution management, and smart cities, providing a feasible solution for building efficient, safe, and reliable smart cities. Therefore, this invention has practical value. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a vision-based real-time vehicle detection method for drone aerial photography in hazy weather, according to an embodiment of this application, is shown.

[0017] Figure 2 The diagram illustrates a novel plug-and-play defogging module FMGRModel for hazy weather, according to an embodiment of this application.

[0018] Figure 3 illustrates the improved strategy for the neck part in the vision-based real-time vehicle detection method for drone aerial photography in hazy weather according to an embodiment of this application, wherein Figure 3(a) abandons the PrFuFPN backbone structure of weighted feature fusion, Figure 3(b) CA attention module, and Figure 3(c) lightweight feature enhancement module DG-CAFM;

[0019] Figure 4 A structural block diagram of a vision-based real-time vehicle detection device for drone aerial photography in hazy weather, according to an embodiment of this application, is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The above description is only an overview of the technical solutions of this application. In order to better understand the technical means of this application and to implement it according to the content of the specification, and to make the above and other objectives, features, and advantages of this application more apparent and understandable, specific embodiments of this application are described below.

[0021] Figure 1 The flowchart of a vision-based drone aerial photography method for real-time vehicle detection in haze, according to an embodiment of this application, is shown, including the following steps:

[0022] Step S110: Obtain a sample image dataset taken by a drone under hazy weather, and extract the vehicle image dataset from the sample image dataset.

[0023] The dataset mentioned above is the public dataset HazyDet, jointly created by researchers from the Army Engineering University of the Chinese People's Liberation Army, Nankai University, Nanjing University of Posts and Telecommunications, and Nanjing University of Science and Technology. This dataset is specifically designed for target detection by UAVs in foggy conditions and includes synthetic images simulating fog effects from natural foggy environments and normal scenes. Simultaneously, it collected a large amount of normal weather data and utilized an Atmospheric Scattering Model (ASM) to simulate and generate large-scale UAV detection data for foggy conditions.

[0024] Step S120: Preprocess the vehicle image dataset and use the preprocessed vehicle image dataset as the training set.

[0025] The training set includes: target vehicle image data and corresponding labels, the labels being used to indicate the location information of the target vehicle.

[0026] Because visual data collected under hazy conditions is prone to confusion, and drone-collected datasets contain a large number of small vehicle targets, the datasets often suffer from missing or incorrect labeling. Therefore, this invention uses the labellmg tool to clean the data, improving vehicle detection accuracy and enhancing the model's robustness. Preprocessing the dataset reduces noise interference, avoids penalties for missing labels, and improves the model's learning efficiency. Simultaneously, dataset preprocessing compensates for vehicle labels partially obscured by haze, enhancing the model's adaptability to hazy scenes.

[0027] Step S130: Train the pre-established improved YOLO target detection model based on the training set to obtain a vehicle target detection model.

[0028] The improved YOLO target detection model is the MgFm-YOLO model.

[0029] First, to address the challenge of blurred and easily confused textures in visual data collected under hazy weather conditions, this invention designs a novel plug-and-play dehazing module, FMGRModel. This module integrates multi-scale fog concentration sensing and physics-driven residual learning methods to process hazy data and can be inserted into the entire improved network to achieve end-to-end learning.

[0030] Secondly, to address the issue of numerous small-scale vehicle targets captured by drones affecting detection accuracy, this invention designs a new structure called PrFmFPN based on the existing Neck architecture. This structure adds a small target detection layer to handle small-sized vehicle targets, and employs cross-scale connections and bidirectional cascading to achieve higher-level feature fusion, thereby improving the overall vehicle detection accuracy.

[0031] Furthermore, this invention addresses the challenge of spatial visual blurring caused by vehicle motion during UAV data collection. It incorporates a CA attention module before PrFmFPN structural feature fusion to increase spatial position sensitivity, and simultaneously designs a lightweight, plug-and-play spatial feature fusion module, DG-CAFM, with both modules working in concert.

[0032] Furthermore, this invention addresses the challenges of real-time requirements and limited hardware platform resources in drone aerial photography. It incorporates a novel S-FocalerIoU loss function, which accelerates convergence by considering orientation alignment and size matching between bounding boxes, while simultaneously focusing on difficult-to-regress samples to increase detection accuracy.

[0033] Finally, on the HayzDet-test dataset, compared with the original YOLOv11s model, mAP@0.5 improved by 6.7%, mAP@0.5:0.95 improved by 10.3%, the model size increased by only 0.8M, and the FPS reached 51.81 (>>30), which fully meets the needs of real-time detection.

[0034] The detailed explanation is as follows:

[0035] This invention addresses the challenges of hazy weather by designing a novel, plug-and-play, lightweight dehazing module, FMGRModel, which can be embedded into the entire object detection network for end-to-end training. This module not only performs dehazing but also enhances the network's focus on target detection within the complex context of hazy weather. Furthermore, end-to-end training significantly improves inference speed compared to the two-stage training approach of dehazing followed by detection. Its structure is shown in the diagram below. Figure 2 As shown. From Figure 2 As can be seen from the data, the FMGRModel defogging module consists of three parts: MSFFB, LGFM, and PRRM. It has three major advantages: lightweight and efficient, multi-scale fog concentration sensing, and physical-driven residual learning.

[0036] MSFFB (Multi-Scale Feature Fusion Block) consists of two parts: Conv, Ghost Conv, and Dilated Conv multi-scale convolutions and skip concatenation. This enables multi-scale feature extraction and cross-level fusion. Conv extracts low-level features (edges / colors), Ghost Conv lightly refines mid-level features, and Dilated Conv extracts high-level semantics with a large receptive field, forming a multi-scale feature extraction pyramid from local to global perspectives. This better captures local details (fog between leaves) and global distribution (dense fog on the skyline). Furthermore, the addition of cross-level feature skip concatenation fuses the original input (low-level RGB information) with deep features (medium-level semantics), preventing information loss while enhancing gradient backflow to avoid gradient vanishing.

[0037] LGFM (Lightweight Ghost Fusion Module) consists of four parts: Conv, Ghost Conv, Tanh activation function, and feature concatenation. It achieves lightweight feature enhancement and multi-level information interaction and fusion. Dual Ghost Conv progressively extracts high-order semantics while reducing computational cost. Furthermore, it forms a cross-level feature fusion pyramid by combining the original input (preserving low-level information), mid-level features (mid-level semantics), and deep features (high-level abstraction), thereby better fusing different levels of fog concentration. Finally, the Tanh activation function is used to compress the fused features to the range [-1, 1], avoiding gradient explosion while preserving negative information.

[0038] PRRM (Physics-aware Residual Refinement Module) consists of three parts: Ghost Conv, deep multiplication, and a physical model, enabling residual fog modeling and physics-driven defogging. It generates a layer of noisy fog using Ghost Conv, and then explicitly models the fog residual (out = x - out), conforming to an atmospheric scattering model.

[0039] (1)

[0040] in For foggy maps, For a clearer image, A represents atmospheric light. Transmittance. Furthermore, by multiplying the deep feature x with the output feature, the fog density map is dynamically modulated, enabling adaptive processing of uneven fog concentrations (such as dense fog in the distance / light fog in the foreground). In high fog regions, the activation value of the deep feature x is large, thus enhancing the residual intensity; in low fog regions, the activation value of the deep feature x is small, thus preserving details.

[0041] In summary, within the entire FMGRModel dehazing module, MSFFB provides multi-scale features, LGFM performs semantic fusion, and PRRM implements physical dehazing. These three components form a complete pipeline to jointly accomplish the dehazing task. Each plays a different role in different scenarios. For dense fog, MSFFB captures the fog distribution with a large receptive field, LGFM enhances the semantics of the fog area, and PRRM provides accurate residual estimation. For light fog, MSFFB preserves fog detail features, LGFM performs multi-level feature fusion, and PRRM suppresses over-smoothing during dehazing.

[0042] This invention designs a novel neck structure, PrFmFPN, to address the complex backgrounds of smog and drone aerial photography. The PrFmFPN structure of this embodiment is shown in Figure 3(a). Based on the original FPN-PAN, this structure employs bidirectional concatenation for nearby feature units and cross-scale connections for more distant feature units, thereby achieving a higher level of feature fusion. At a deeper level, bidirectional concatenation can simultaneously map semantic information from shallow features to deep features and localization information from deep features to shallow features, achieving a high-level bidirectional fusion that avoids feature loss and improves the accuracy of vehicle detection at different scales. Cross-scale connections can fuse more feature information, especially some original feature information, without significantly increasing costs.

[0043] Furthermore, while weighted feature fusion can dynamically adjust feature ratios through weight learning, insufficient weight learning can lead to the suppression of key features. Additionally, when weights approach zero, it weakens gradient backpropagation of deep features, affecting training stability. Therefore, this embodiment abandons weighted feature fusion and adopts the original Concat fusion.

[0044] Furthermore, PrFmFPN adds a Coordinate Attention (CA) module to the first half of the entire structure, as shown in Figure 3(b). The CA module can explicitly preserve spatial location information through X / Y coordinate decomposition and pooling, while the generated attention weights contain joint spatial-channel information, which can dynamically suppress noise in high fog concentration areas and enhance vehicle edge features in low fog areas.

[0045] In the latter half of the PrFmFPN structure, this invention introduces a novel plug-and-play lightweight feature enhancement module, DG-CAFM (Dual-Ghost Coordinate Attention Feature Module), as shown in Figure 3(c). The DG-CAFM module consists of three parts: Ghost Conv, CA attention, and residual connections, achieving synergistic optimization of lightweight feature enhancement and accurate position awareness in complex backgrounds. The first layer, Ghost Conv, generates ghost features through linear transformation, reducing computation while initially extracting low-frequency contour features under haze conditions. A 5x5 large kernel is used to enhance the contextual awareness of blurred targets. The CA attention module locates key regions through coordinate attention, enhancing vehicle edge and texture channel responses, suppressing background noise caused by residual haze, and dynamically repairing spatial sensitivity that may be lost due to lightweighting. The second layer, Ghost Conv, reconstructs local high-resolution features using a 3x3 kernel, fusing multi-scale information from attention weights, and retaining clear features after dehazing with residual connections. Overall, it implements a three-stage process of coarse feature extraction, position calibration, and fine reconstruction, enhancing feature information.

[0046] Overall, it is feasible for PrFmFPN to use the CA attention module in the first half and the DG-CAFM feature enhancement module in the second half. CA attention provides position-sensitive calibration of low-level features, enhancing local features of vehicle edges / textures and providing geometrically well-defined input features for DG-CAFM. The DG-CAFM module performs robust reconstruction of high-level features to remove residual fog and performs multi-scale fusion, and utilizes the CA-calibrated features for fine-tuning. The two modules work together: the first-half CA eliminates redundant computation of Ghost Conv, while the lightweight nature of the second-half DG-CAFM offsets the computational burden of deep, large feature maps; the first-half CA effectively preserves vehicle details (preventing premature blurring by Ghost Conv), while the second-half DG-CAFM achieves more thorough defogging in deep, large receptive fields.

[0047] In complex environments such as smoggy weather and drone aerial photography, the original simple CIOU loss function is insufficient to meet the actual target detection requirements. Therefore, this invention fuses the SIOU loss function with the FocalerIoU loss function to obtain a new loss function, S-FocalerIOU.

[0048] The SIOU loss function consists of four parts: IOU loss, angle cost, distance cost, and shape cost.

[0049] Angular cost:

[0050] (2)

[0051] in It is the distance from the center point. These are the coordinates of the center of the prediction box.

[0052] Distance cost:

[0053] (3)

[0054] in From angle cost Decision: .

[0055] Shape cost:

[0056] (4)

[0057] The SIOU loss function is:

[0058] (5)

[0059] Secondly, the FocalerIoU loss function focuses on hard samples by dynamically scaling the IOU threshold, and its formula is shown in formula (6).

[0060] (6)

[0061] From this, we can derive the S-FocalerIoU loss function, the formula of which is shown in formula (7).

[0062] (7)

[0063] Finally, after deriving all the formulas, the following section will explain in detail why the S-FocalerIoU loss function was designed by fusing SIOU loss and FocalerIOU loss. First, the angular cost of the SIOU loss function can enhance positioning accuracy through angle alignment to handle boundary blurring caused by haze. The distance cost, considering the distance to the center point, can compensate for positioning difficulties under low contrast. The shape cost can maintain the consistency of the vehicle's aspect ratio, even if some vehicles are obscured by haze, thereby accelerating convergence and improving positioning accuracy (suitable for real-time UAV detection). FocalerIOU, by introducing dynamic threshold scaling, focuses on samples that are difficult to regress (such as small targets and targets obscured by haze) and alleviates the imbalance between vehicles and the background in UAV aerial photography.

[0064] In summary, by designing the S-FocalerIoU loss function, we can achieve dual optimization of SIoU geometric constraints and FocalerIoU sample weight optimization; it is more robust to blurring and occlusion caused by haze; and it combines the convergence characteristics of SIoU's angle cost accelerating initial convergence and FocalerIoU's mid-term focused and stable training.

[0065] Step S140: Input the target vehicle image data under hazy weather into the vehicle target detection model to detect the location information of the target vehicle.

[0066] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Rather, the invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide the public with a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0067] Figure 4 The diagram illustrates a structural block diagram of a vision-based real-time vehicle detection device for drone aerial photography in hazy weather, according to an embodiment of this application. The detection device includes:

[0068] The S210 data acquisition module is used to acquire sample image datasets under hazy weather conditions in real time through drone equipment, and extract vehicle image datasets from the sample image datasets.

[0069] The S220 data preprocessing module is used to preprocess the vehicle image dataset and use the preprocessed vehicle image dataset as a training set. The data preprocessing module is connected to the data acquisition module and the training module respectively.

[0070] The S230 training module is used to train a pre-established improved YOLO object detection model based on the training set to obtain a vehicle object detection model.

[0071] The S240 detection module is used to input the image data of the target vehicle under hazy weather into the vehicle target detection model to detect the location information of the target vehicle.

[0072] The detection device and detection method provided in the above-described device embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0074] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A visual-based real-time detection method for haze unmanned aerial vehicle (UAV) aerial vehicle, characterized in that, The method comprises the following steps: Step 1: obtaining a sample image dataset taken by an unmanned aerial vehicle in a smoggy weather, and extracting a vehicle image dataset from the sample image dataset; Step 2: pre-processing the vehicle image dataset, and taking the pre-processed vehicle image dataset as a training set; the training set comprises target vehicle image data and corresponding labels, wherein the labels are used to indicate position information of the target vehicle; Step 3: training a pre-established improved YOLO target detection model based on the training set, to obtain a vehicle target detection model, wherein the improved YOLO target detection model is an MgFm-YOLO model; Step 4: inputting target vehicle image data to be detected in a smoggy weather into the vehicle target detection model, to detect position information of the target vehicle.

2. The identification method according to claim 1, characterized in that, The pre-processing of the vehicle image dataset comprises the following steps: Step 1: obtaining a public dataset HazyDet; Step 2: pre-processing data of the public dataset HazyDet, wherein the data pre-processing comprises using a labellmg tool to clean up mislabeling and missing labeling of small-size vehicle images in the vehicle image dataset, so as to improve detection accuracy of the vehicle and robustness of the model.

3. The identification method according to claim 1, characterized in that, The improved YOLO target detection model comprises: a dehazing module FMGRModel, which is added to a backbone part, and is used to further blur edges and textures in the case of color confusion between a background and a target in vehicle image data collected in a smoggy weather; a PrFmFPN structure, which replaces an original neck part, and is used to solve the problem that small-size vehicle targets are mostly in vehicle data collected by an unmanned aerial vehicle working at a height of 50-200 meters; an S-FocalerIoU loss function, which is a bounding box regression loss function, and is used to accelerate convergence and focus on difficult regression samples.

4. A visual-based haze unmanned aerial vehicle aerial vehicle real-time detection device, characterized in that, The detection device comprises: a data acquisition module, which is used to obtain a sample image dataset taken by an unmanned aerial vehicle in a smoggy weather, and extract a vehicle image dataset from the sample image dataset; a data pre-processing module, which is used to pre-process the vehicle image dataset, and take the pre-processed vehicle image dataset as a training set; the training set comprises target vehicle image data and corresponding labels, wherein the labels are used to indicate position information of the target vehicle; a training module, which is used to train a pre-established improved YOLO target detection model based on the training set, to obtain a vehicle target detection model, wherein the improved YOLO target detection model is an MgFm-YOLO model; a detection module, which is used to input target vehicle image data to be detected in a smoggy weather into the vehicle target detection model, to detect position information of the target vehicle.

Images