A Method and System for Small Target Detection in UAV Aerial Photography Based on Multi-Path Fusion Sensing

Abstract

This invention relates to a method and system for small target detection in UAV aerial photography based on multi-path fusion perception. The method includes the following steps: Step 1, obtaining a trained and inferable detection model; Step 2, obtaining accurate small target detection results at the edge; Step 3, obtaining a closed loop for the entire process of small target detection and monitoring in UAV aerial photography. The beneficial effects of this invention are: 1. It solves the problem of coordinate information loss and feature ambiguity of small targets under current UAV aerial photography perspectives due to multiple downsampling, while generating high-quality detection results with accurate positioning and semantic alignment in complex aerial photography backgrounds; 2. It constructs an edge computing acceleration module, realizing real-time deployment of the algorithm at the edge; 3. It constructs an asynchronous visualization monitoring module; 4. It improves the small target recall rate and real-time performance of human-computer interaction in dynamic and complex scenes of UAV aerial photography detection systems.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a method and system for detecting small targets in UAV aerial photography based on multi-path fusion perception. Background Technology

[0002] With the rapid development of drone technology, drone aerial photography has been widely used in urban traffic monitoring, power line inspection, agricultural and forestry plant protection, and disaster relief. Currently, deep learning-based object detection algorithms (such as the YOLO series, SSD, Faster R-CNN, etc.) have achieved significant detection results in regular natural scene images (such as the PASCAL VOC and COCO datasets). These algorithms typically extract image features through convolutional neural networks and utilize feature pyramid networks (FPNs) for multi-scale prediction. However, directly applying existing general object detection algorithms to drone aerial photography scenarios still presents the following major technical challenges:

[0003] First, drone aerial photography offers a high and wide field of view, resulting in ground targets occupying a very small proportion of the image. In existing deep convolutional neural networks, as the number of network layers increases, multiple pooling and downsampling operations easily filter out the geometric details and texture information of small targets, making them unable to be effectively represented in deep feature maps, thus causing serious missed detections.

[0004] Secondly, aerial images typically have extremely complex backgrounds, containing a large amount of interfering information such as buildings, vegetation, and shadows. Existing technologies often struggle to distinguish small targets from similar textures in complex backgrounds, easily mistaking background noise for targets. Furthermore, a single feature extraction path often focuses on local information while ignoring global contextual information, leading to a decrease in detection capability when the target is partially occluded.

[0005] In summary, designing a detection model that can effectively preserve the features of small targets, suppress background interference, and integrate perception information through multi-path fusion, along with a system for visual monitoring and status feedback, is a pressing technical challenge in the field of UAV vision. Summary of the Invention

[0006] To address the technical problem of feature information mismatch (or misalignment) in existing small target detection technologies, this invention provides a method and system for small target detection in UAV aerial photography based on multi-path fusion perception. It proposes a multi-dimensional feature fusion module (DCFM module), which is configured to integrate coordinate information, spatial location information, and semantic information in multiple dimensions. This invention aims to solve the following technical problems:

[0007] 1. Address the issue of weak perception of small targets in complex aerial photography backgrounds;

[0008] 2. Solve the problem of difficult real-time deployment of complex detection models at the edge of drones;

[0009] 3. Address the issue of the lack of real-time visual monitoring and status feedback mechanisms in aerial photography inspection systems.

[0010] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for detecting small targets in UAV aerial photography based on multi-path fusion perception, comprising the following steps:

[0011] Step 1: Using YOLOv11s as the baseline model, a dual-coordinate space semantic fusion module is introduced, and a lightweight detection model framework is built by combining multi-path fusion and attention enhancement mechanisms; the model is trained and optimized on the drone aerial photography small target dataset, and the model iteration of multi-dimensional feature fusion is completed to obtain a trained detection model that can be used for inference.

[0012] Step 2: Based on the detection model obtained in Step 1, the model is deployed and optimized for inference on the Xiangchengpai AI PRO edge computing unit through four steps: format conversion and graph optimization, mixed precision quantization and compilation, inference engine construction, post-processing and result mapping. Real-time inference calculations are performed on drone aerial images / video streams to obtain accurate detection results including target category, location coordinates and confidence level.

[0013] Step 3: Based on the accurate edge detection results obtained in Step 2, synchronously collect the operating status data of the airborne hardware equipment, complete the two-way data interaction between the airborne terminal and the ground station through the communication transmission module, and then realize the rendering and display of the detection results, category filtering, multi-channel monitoring and real-time feedback of equipment status by the ground station's visual monitoring terminal. Combined with the human-machine interaction interface, remote control of the detection process is realized, forming a closed loop of drone aerial photography small target detection and monitoring.

[0014] Based on the above technical solution, the present invention can be further improved as follows:

[0015] Furthermore, step 1 specifically involves:

[0016] Step 1.1: First, build the detection model architecture and input the feature map into the feature enhancement module to improve the feature representation of small targets;

[0017] Step 1.2 involves performing channel attention and spatial attention processing through dual-channel feature separation, combining the network layers to complete multi-path feature fusion, and using a drone aerial photography small target dataset to complete model training, validation, and optimization, ultimately obtaining a trained detection model that can be used for inference.

[0018] Furthermore, step 2 specifically involves:

[0019] Step 2.1, Model Format Conversion and Graph Optimization: Export the detection model obtained in Step 1 to ONNX format, fuse convolution, batch normalization, and activation function layers, remove redundant nodes during the training phase, and standardize the custom operators of the dual-coordinate space semantic fusion module.

[0020] Step 2.2, Hybrid Precision Quantization and Compilation: Convert the ONNX format model into an inference file adapted to Orange Pi, select typical aerial images of small targets for calibration, use INT8 quantization for the backbone network, standardize the dual coordinate space semantic fusion module and the detection head to retain FP16 precision, and compile to generate a heterogeneous computation graph adapted to the Orange Pi architecture.

[0021] Step 2.3, Edge Inference Engine Construction: Based on the C++ / Python engine, the underlying API is called to load the compiled model. Zero-copy memory allocation is used to reduce data copy latency. RGA hardware / CPU multi-threading is used to scale, normalize and perform other preprocessing on the drone aerial video stream before inputting it into the model.

[0022] Step 2.4, Post-processing and Result Mapping: The NPU performs parallel inference on the pre-processed aerial data, outputting target category probability and position offset tensors. Then, the CPU executes the non-maximum suppression algorithm to decode the relative offset into the absolute coordinates of the original image, filtering out low-confidence and overlapping redundant candidate boxes, and finally obtaining accurate small target detection results at the edge.

[0023] Furthermore, step 3 specifically involves:

[0024] Step 3.1, Data Acquisition and Encoding Transmission: The airborne terminal synchronously acquires the detection results from Step 2, as well as NPU utilization, CPU temperature, and other device status data. After compressing and encoding the video stream and data, the detection results are transmitted through the Socket interface, and the device status data packets are broadcast to the ground station via the UDP protocol.

[0025] Step 3.2, Ground station data reception and processing: The ground station visual monitoring terminal receives and decodes all data transmitted from the airborne terminal. The multi-source data management module completes the data source adaptation, and the asynchronous inference engine completes image decoding and data post-processing through an independent working thread. The processing results are asynchronously transmitted to the main interface thread through the signal and slot mechanism.

[0026] Step 3.3, Visualization and Intelligent Filtering of Detection Results: The rendering layer overlays the target bounding box, category label and confidence score on the original aerial video stream. The semantic category filtering module performs logical filtering on the detection results of Step 2 according to user needs, only displaying the filtered targets and statistically analyzing the category distribution in real time.

[0027] Step 3.4, Hardware Status Monitoring and Human-Machine Interaction: The ground station analyzes the status data of the airborne equipment, visualizes the hardware operation status through the instrument panel, completes the abnormal warning through the built-in threshold comparison logic, and provides human-machine interaction interfaces such as confidence threshold adjustment and detection mode switching to realize remote control of the detection process and form a complete detection and monitoring closed loop.

[0028] This invention also provides a UAV aerial photography small target detection system based on multi-path fusion perception, for applying the above-mentioned UAV aerial photography small target detection method based on multi-path fusion perception, including:

[0029] The airborne edge computing subsystem, mounted on an unmanned aerial vehicle (UAV) platform, integrates all the hardware and software resources of the edge inference subsystem. It provides physical support and operating environment for image acquisition, model inference, and status acquisition at the edge. At the same time, it serves as the airborne data source for the communication transmission module, enabling the unified collection and external transmission of detection data and equipment status data.

[0030] The communication transmission module connects the airborne edge computing subsystem and edge inference subsystem with the ground station's visual monitoring terminal, enabling two-way data interaction. It is responsible for the real-time encoding, compression, and transmission of aerial video streams, target detection results, and airborne equipment status data. At the same time, it transmits human-machine interaction control commands from the ground station back to the airborne terminal, ensuring the real-time performance and integrity of data transmission and the accurate delivery of commands.

[0031] The edge inference subsystem, deployed on the Orange Pie AI PRO hardware platform of the airborne edge computing subsystem, is responsible for the acquisition, preprocessing, small target detection model inference, and post-processing of detection results of drone aerial images / video streams. At the same time, it collects real-time operating status data of the airborne hardware and is the core functional unit for realizing real-time small target detection at the edge.

[0032] The visual monitoring terminal, deployed at the ground station, is developed using Python based on a client-server architecture and a cross-platform graphical interface framework. It is responsible for receiving and decoding all data transmitted by the communication transmission module, realizing the visualization rendering of detection results, multi-source data management, intelligent category filtering, and multi-channel parallel monitoring. At the same time, it monitors the status of airborne hardware in real time and provides anomaly warnings. It provides a visual human-machine interaction control interface, allowing ground operators to intuitively grasp the detection process and results.

[0033] Based on the above technical solution, the present invention can be further improved as follows:

[0034] Furthermore, the edge inference subsystem includes:

[0035] The image acquisition unit acquires high-resolution, real-time drone aerial video streams, providing raw input data for model inference and serving as the data source entry point for edge inference.

[0036] The NPU inference acceleration unit loads a lightweight detection model compiled with mixed precision quantization, completes parallel inference operations on the aerial video stream, and outputs the raw tensor data of the target detection. It is the core computing unit for edge inference.

[0037] The telemetry data broadcasting unit collects real-time operational status data of the airborne hardware, packages it according to specifications, and outputs it to the communication transmission module, providing data support for the hardware status monitoring of the ground station's visual monitoring terminal.

[0038] Furthermore, the visual monitoring terminal includes:

[0039] The multi-source data management module enables flexible switching and unified management of system input data sources, adapts to various data input scenarios of drone aerial photography, and provides a stable and standardized data source for subsequent inference post-processing and visualization.

[0040] The asynchronous inference engine separates interface rendering from background data processing tasks, avoiding time-consuming decoding and post-processing operations that cause terminal interface lag, and ensuring the real-time performance and smoothness of visual monitoring.

[0041] The rendering layer module intuitively overlays the accurate detection results output by the edge inference subsystem onto the original aerial video stream, realizing a visual display of the detection results, allowing operators to clearly understand the location, category, and confidence level of the target;

[0042] The semantic category filtering module solves the problem of information overload caused by dense targets and numerous categories in drone aerial photography scenarios, enabling accurate filtering of detection results and improving the operator's visual search efficiency;

[0043] The multi-channel parallel monitoring module supports simultaneous monitoring of multiple aerial video streams, adapting to application scenarios involving multiple drone cameras or collaborative detection by multiple drones, and enabling parallel visualization of multiple video streams and corresponding detection results.

[0044] Furthermore, the semantic category filtering module includes:

[0045] The preset scheme management unit stores a category whitelist for specific application scenarios. The preset scenarios include personnel reconnaissance, vehicle statistics, and traffic flow monitoring. Operators can directly select the appropriate preset scheme.

[0046] The post-processing filtering unit receives all detection box data output by the edge inference subsystem, and performs logical AND filtering operations on the detection results based on the user's currently active category whitelist or keywords entered in the search box to remove the target data that did not match.

[0047] The dynamic rendering unit only visualizes and draws the targets filtered by the post-processing filtering unit, while simultaneously calculating the number and category distribution of the filtered targets in real time and displaying the statistical results intuitively on the terminal interface.

[0048] The beneficial effects of this invention are:

[0049] 1. This paper addresses the issues of coordinate information loss and feature blurring caused by multiple downsampling of small targets under the current drone aerial photography perspective. At the same time, a multi-path fusion perception mechanism is introduced into the feature extraction network. Contextual information is captured through dual-path pooling and deep and shallow features are aligned using position encoding. Combined with a hybrid precision quantization strategy, high-quality detection results with accurate positioning and semantic alignment are finally generated in complex aerial photography backgrounds.

[0050] 2. An edge computing acceleration module was built, which utilizes a zero-copy memory sharing mechanism and heterogeneous computing architecture to control the inference latency and memory usage of the detection model on resource-constrained hardware such as Orange Pie AI PRO, and realizes real-time deployment of the algorithm at the edge.

[0051] 3. An asynchronous visual monitoring module was built, which decouples the interface rendering and background inference based on Qt multithreading technology, and combines UDP telemetry mechanism to provide real-time feedback on hardware load status;

[0052] 4. Improved the recall rate of small targets and the real-time performance of human-computer interaction in dynamic and complex scenarios of the UAV aerial photography detection system. Attached Figure Description

[0053] Figure 1 This is a diagram of the DCFM-YOLO architecture.

[0054] Figure 2 This is a block diagram of the ground station visualization monitoring subsystem of the system described in this invention;

[0055] Figure 3 This is a comparison chart of heat maps. Detailed Implementation

[0056] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0057] A method for detecting small targets in UAV aerial photography based on multi-path fusion perception includes the following steps:

[0058] Step 1: Using YOLOv11s as the baseline model, a dual-coordinate spatial semantic fusion module (DCFM) is introduced. A lightweight detection model framework is built by combining multi-path fusion and attention enhancement mechanisms. The model is trained and optimized on a small target dataset captured by drone aerial photography, completing iterative model iterations to achieve multi-dimensional feature fusion, resulting in a trained DCFM-YOLO detection model suitable for inference. Details are as follows:

[0059] Step 1.1: First, build the DCFM-YOLO model architecture and input the feature map into the feature enhancement module to improve the feature representation of small targets.

[0060] Step 1.2 involves performing channel attention and spatial attention processing through dual-channel feature separation (x1 (dim / 4), x2 (3*dim / 4)). Multi-path feature fusion is then achieved by combining the DAFM module, Bottleneck, C3k2, C2PSA and other network layers. The model is trained, validated and optimized using a small target dataset captured by UAV aerial photography, and finally the trained DCFM-YOLO detection model is obtained.

[0061] In the field of computer vision, small object detection has always been a highly challenging problem. Existing feature extraction modules typically employ a single feature extraction path, which fails to fully capture the multi-scale features of small objects, and coordinate information is easily lost and difficult to match in deep networks. Step 1 addresses the triple challenges of coordinate information loss, insufficient feature representation, and information mismatch in small object detection by proposing a dual coordinate space semantic fusion module (DCFM), the structure of which is shown in the figure below. Figure 1 As shown in the upper middle section, the feature map first enhances feature expression through a feature enhancement module, and then effectively integrates spatial information, deep semantic features and coordinate position information in UAV aerial images through a dual-channel feature separation and multi-path fusion strategy, achieving unified optimization of accuracy, efficiency and versatility.

[0062] Step 1 addresses the shortcomings of general detection algorithms that rely on a single feature extraction path. Positional encoding aligns shallow and deep features, preventing coordinate information loss and insufficient feature representation for small targets due to multiple downsampling. A multi-path fusion mechanism simultaneously captures local details and global contextual information, while a lightweight architecture adapts to edge deployment requirements. The trained model achieves improved detection accuracy while maintaining a lightweight design, effectively preserving small target features, suppressing interference from complex aerial backgrounds, and addressing the weakness in small target perception. This provides a directly usable foundational model for subsequent edge deployments.

[0063] Step 2: Based on the DCFM-YOLO detection model trained in Step 1, the model is deployed and optimized for inference on the OrangePie AI PRO edge computing unit through four steps: format conversion and graph optimization, mixed-precision quantization and compilation, inference engine construction, and post-processing and result mapping. Real-time inference computation is then performed on drone aerial images / video streams to obtain accurate detection results including target category, location coordinates, and confidence level. To address the technical challenge of complex detection models not being able to run in real-time on resource-constrained drone onboard devices, the aforementioned small target detection model containing the DCFM module is deployed on the OrangePie AI PRO edge computing unit. Details are as follows:

[0064] Step 2.1, Model Format Conversion and Graph Optimization: Export the trained model obtained in Step 1 to ONNX format, fuse convolutional layers, batch normalization, and ReLU activation function layers, remove redundant nodes from the training phase, and standardize the custom operators of the DCFM module. First, export the trained deep learning model to an intermediate representation format (ONNX format). During this process, graph optimization operations are performed:

[0065] Operator fusion: The convolutional layers, batch normalization layers and activation function layers (ReLU) in the model are fused to reduce the number of memory accesses during inference.

[0066] Redundant node removal: Remove nodes used only during the training phase and standardize the custom operators in the proposed DCFM module to ensure they conform to the general operator definition for subsequent toolchain recognition.

[0067] Step 2.2, Hybrid Precision Quantization and Compilation: The ONNX format model is converted into an inference file (.om file) adapted to Orange Pi. Typical aerial images of small targets are selected for calibration. The backbone network is quantized using INT8, while the DCFM module and detector head retain FP16 precision. Compilation generates a heterogeneous computation graph adapted to the Orange Pi architecture. To accelerate performance on the Orange Pi AI PRO's NPU, a hybrid precision quantization strategy is adopted:

[0068] Calibration: Select typical images from the aerial small target validation set as the calibration dataset, input them into the model for forward propagation, and statistically analyze the activation value distribution (Kl divergence or maximum value) of each layer feature map.

[0069] Precision allocation: For layers with relatively uniform parameter distribution, such as the backbone network, INT8 quantization is used to maximize inference speed; for the core DCFM module and the final detection head of this invention, in order to preserve the coordinate accuracy and weak features of small targets, FP16 precision is used for calculation.

[0070] Compilation and Generation: Finally, a heterogeneous computing graph adapted to the OrangePie AI PRO architecture is generated.

[0071] Step 2.3, Edge Inference Engine Construction: An engine is built based on C++ / Python. It calls the underlying API to load the pre-compiled model, employs zero-copy memory allocation to reduce data copy latency, and utilizes RGA hardware / CPU multi-threading to perform preprocessing such as scaling and normalization on the drone aerial video stream before inputting it into the model. In the Orange Pi embedded environment, the inference engine is built based on C++ / Python. This engine loads the pre-compiled model by calling the underlying API and performs the following operations:

[0072] Zero-copy memory allocation: Input and output buffers are allocated directly in the shared memory area between the NPU and the CPU, avoiding repeated copying of image data in different memory spaces and reducing inference latency.

[0073] Multi-threaded preprocessing: Utilize OrangePi's RGA hardware or CPU multi-threading to scale, normalize, and convert the color space of the input drone video stream to match the model's input requirements.

[0074] Step 2.4, Post-processing and Result Mapping: The NPU performs parallel inference on the preprocessed aerial data, outputting a tensor containing the target class probability and position offset. The CPU then executes the Non-Maximum Suppression (NMS) algorithm to decode the relative offsets into the absolute coordinates of the original image, filtering out low-confidence and overlapping redundant candidate boxes, ultimately obtaining accurate small target detection results at the edges. After inference, the NPU outputs a tensor containing the target class probability and position offset. Because the NPU excels at matrix operations but not logical judgments, the system utilizes the CPU to execute the Non-Maximum Suppression (NMS) algorithm.

[0075] Coordinate decoding: Maps the relative offsets output by the model back to the absolute coordinates of the original aerial image.

[0076] Threshold filtering: Remove candidate boxes with confidence scores below a preset threshold and eliminate redundant boxes with excessive overlap.

[0077] The final test results are transmitted in real time to the QT visual monitoring terminal via the Socket communication interface.

[0078] Step 2 addresses the limited hardware resources at the edge by reducing computational load through operator fusion and redundancy removal. Hybrid precision quantization improves inference speed while maintaining the detection accuracy of core modules. Zero-copy memory sharing and hardware-accelerated preprocessing enable efficient collaboration across heterogeneous computing architectures, and the combination of NPU parallel inference and CPU logic post-processing fully leverages hardware advantages. This solves the problem of complex detection models not being able to be deployed in real time at the UAV edge, significantly reducing inference latency and GPU memory usage, enabling real-time parallel inference of the detection model on the airborne end, and accurately mapping the output detection results to the original aerial images, providing core data support for subsequent ground station visualization monitoring.

[0079] Step 3: Based on the accurate edge detection results obtained in Step 2, synchronously collect the operational status data of the airborne hardware equipment. Two-way data interaction between the airborne terminal and the ground station is completed through the communication transmission module. The ground station's visual monitoring terminal then renders and displays the detection results, filters categories, performs multi-channel monitoring, and provides real-time feedback on equipment status. Combined with the human-machine interface, remote control of the detection process is achieved, forming a closed loop for the entire process of UAV aerial small target detection and monitoring. Details are as follows:

[0080] Step 3.1, Data Acquisition and Encoding Transmission: The airborne terminal synchronously acquires the detection results from Step 2 and device status data such as NPU utilization and CPU temperature. After compressing and encoding the video stream and data, the detection results are transmitted through the Socket interface, and the device status data packets are broadcast to the ground station via the UDP protocol.

[0081] Step 3.2, Ground Station Data Reception and Processing: The ground station visual monitoring terminal receives and decodes all data transmitted from the airborne terminal. The multi-source data management module completes the data source adaptation, and the asynchronous inference engine completes image decoding and data post-processing through an independent working thread. The processing results are asynchronously transmitted to the main interface thread through the signal and slot mechanism.

[0082] Step 3.3, Visualization and Intelligent Filtering of Detection Results: The rendering layer overlays the target bounding box, category label and confidence score on the original aerial video stream. The semantic category filtering module performs logical filtering on the detection results of Step 2 according to user needs, only displaying the filtered targets and statistically analyzing the category distribution in real time.

[0083] Step 3.4, Hardware Status Monitoring and Human-Machine Interaction: The ground station analyzes the status data of the airborne equipment, visualizes the hardware operation status through the instrument panel, completes the abnormal warning through the built-in threshold comparison logic, and provides human-machine interaction interfaces such as confidence threshold adjustment and detection mode switching to realize remote control of the detection process and form a complete detection and monitoring closed loop.

[0084] Step 3 employs a client-server architecture to decouple the airborne terminal from the ground station. Asynchronous multithreading technology separates interface rendering from background data processing, preventing interface lag. A UDP telemetry mechanism enables real-time awareness of device status, a semantic filtering mechanism addresses the information overload problem in aerial inspection, and bidirectional communication enables remote control of the inspection process. This solves the problem of the lack of real-time visual monitoring and status feedback in aerial inspection systems, maintaining a smooth refresh rate on the main interface and fast button response time. It achieves accurate filtering of inspection categories and parallel monitoring of multiple video streams, allowing ground operators to monitor inspection results and airborne hardware status in real time, providing timely warnings of equipment overheating, excessive load, and other issues. This completes the entire closed-loop process from edge inference to ground station monitoring, improving the practicality and interactivity of the inspection system.

[0085] This invention also provides a UAV aerial small target detection system based on multi-path fusion perception, used to apply the aforementioned UAV aerial small target detection method based on multi-path fusion perception. This system mainly consists of an airborne edge computing subsystem (backend) and a ground station visualization monitoring subsystem (frontend, such as...). Figure 2 It consists of two parts (as shown), specifically including:

[0086] 1. Airborne Edge Computing Subsystem

[0087] As the hardware carrier and core deployment unit of the system, it is mounted on the UAV platform and integrates all the hardware and software resources of the edge inference subsystem. It provides physical support and operating environment for image acquisition, model inference and status acquisition at the edge. At the same time, as the airborne data source of the communication transmission module, it realizes the unified collection and external transmission of detection data and equipment status data.

[0088] 2 Communication Transmission Module

[0089] As the system's "data bridge," it connects the airborne edge computing subsystem and edge inference subsystem with the ground station's visual monitoring terminal, enabling two-way data interaction. It is responsible for the real-time encoding, compression, and transmission of aerial video streams, target detection results, and airborne equipment status data. Simultaneously, it transmits human-machine interaction control commands from the ground station back to the airborne terminal, ensuring the real-time performance and integrity of data transmission and the accurate delivery of commands.

[0090] 3 Edge Inference Subsystem

[0091] As the core inference computing unit of the system, it is deployed on the OrangePie AI PRO hardware platform of the airborne edge computing subsystem. It is responsible for acquiring, preprocessing, inferring small target detection models, and post-processing detection results from drone aerial images / video streams. Simultaneously, it collects real-time operational status data from the airborne hardware, making it the core functional unit for achieving real-time small target detection at the edge. The edge inference subsystem specifically includes:

[0092] 3.1 Image Acquisition Unit

[0093] It acquires high-resolution, real-time drone aerial video streams, providing raw input data for model inference and serving as the data source entry point for edge inference;

[0094] 3.2 NPU Inference Acceleration Unit

[0095] It loads a lightweight detection model compiled with mixed precision quantization, completes parallel inference operations on aerial video streams, and outputs raw tensor data of target detection. It is the core computing unit for edge inference.

[0096] 3.3 Telemetry Data Broadcasting Unit

[0097] It collects real-time operational status data of airborne hardware, packages it according to specifications, and outputs it to the communication transmission module to provide data support for hardware status monitoring of the ground station's visual monitoring terminal.

[0098] 4 Visual monitoring terminal

[0099] Deployed at the ground station, it serves as the core of the system's human-machine interaction and monitoring. Developed based on a client-server (C / S) architecture and Qt for Python (a Python version of a cross-platform graphical interface framework), it is responsible for receiving and decoding all data transmitted by the communication module. It enables visualized rendering of detection results, multi-source data management, intelligent category filtering, and multi-channel parallel monitoring. Simultaneously, it monitors the onboard hardware status in real time and provides anomaly warnings. It offers a visualized human-machine interaction control interface, allowing ground operators to intuitively grasp the detection process and results. The visualized monitoring terminal specifically includes:

[0100] 4.1 Multi-source data management module

[0101] It enables flexible switching and unified management of system input data sources, adapts to various data input scenarios of drone aerial photography, and provides a stable and standardized data source for subsequent inference post-processing and visualization;

[0102] 4.2 Asynchronous Inference Engine

[0103] Its separation of interface rendering and background data processing tasks avoids time-consuming decoding and post-processing operations that cause terminal interface lag, ensuring the real-time performance and smoothness of visual monitoring.

[0104] 4.3 Rendering Layer Module

[0105] It intuitively overlays the accurate detection results output by the edge inference subsystem onto the original aerial video stream, realizing a visual display of the detection results, allowing operators to clearly understand the location, category, and confidence level of the target;

[0106] 4.4 Semantic Category Filtering Module

[0107] It addresses the information overload problem caused by densely packed and diverse targets in drone aerial photography scenarios, enabling precise filtering of detection results and improving the operator's visual search efficiency. The semantic category filtering module specifically includes:

[0108] 4.4.1 Preset Scheme Management Unit

[0109] It stores category whitelists for specific application scenarios, with preset scenarios including personnel reconnaissance, vehicle statistics, and traffic flow monitoring. Operators can directly select the appropriate preset scheme.

[0110] 4.4.2 Post-processing filtration unit

[0111] It receives all detection box data output by the edge inference subsystem, and performs a logical AND filtering operation on the detection results based on the user's currently active category whitelist or keywords entered in the search box (Chinese / English are both acceptable) to remove the target data that did not match.

[0112] 4.4.3 Dynamic Rendering Unit

[0113] It only visualizes the targets filtered by the post-processing filtering unit, and simultaneously calculates the number and category distribution of the filtered targets in real time, and displays the statistical results intuitively on the terminal interface.

[0114] 4.5 Multi-channel parallel monitoring module

[0115] It supports simultaneous monitoring of multiple aerial video streams, adapts to application scenarios of multi-camera or multi-drone collaborative detection, and enables parallel visualization of multiple video streams and corresponding detection results.

[0116] The following section details a typical implementation process of the present invention to help better understand how to apply the above-mentioned invention to practical operations. To objectively and comprehensively evaluate the performance of the proposed model (DCFM-YOLO), the present invention conducted a series of detailed comparative experiments on the highly challenging VisDroneDET2019 dataset, comparing the DCFM-YOLO model with current mainstream lightweight UAV target detection models, as shown in Table 1.

[0117] Table 1 Comparative Experiments

[0118]

[0119] The experimental data in Table 1 show that the model (ours) proposed in this invention has achieved the best performance in all key detection accuracy indicators, with an accuracy of 53.3%, a recall of 42.0%, an average accuracy of 42.8%, a parameter count of 2.7M, and a floating-point count of 21.4G.

[0120] To verify the effectiveness of the proposed DCFM module in feature extraction, we visualized the feature maps of the intermediate layers of the network, as shown below. Figure 3 As shown in -a, when processing aerial images using existing general detection algorithms, the network's attention mechanism is rather divergent due to the small pixel proportion of small targets and the complex background. Feature heatmaps show that high-response areas not only cover target vehicles but also incorrectly cover parts of the road surface and roadside vegetation. Furthermore, the feature activation strength for distant, small targets is weak, easily leading to missed detections. In contrast, as... Figure 3 As shown in -b, after introducing the multi-path fusion perception module proposed in this invention, the feature heatmap exhibits significant focusing characteristics. High-response hotspots are precisely concentrated at the geometric center of the vehicle to be detected, highly overlapping with the target's bounding box. The feature responses of complex background regions are significantly suppressed, proving that the DCFM module effectively enhances the alignment capability between semantic features and spatial information through spatial location encoding. For extremely small targets at image edges or in the distance, the algorithm of this invention can still maintain a high feature activation intensity, verifying the effective preservation and enhancement of feature information of small targets by the multi-path fusion mechanism.

[0121] To verify the effectiveness of the proposed visualization monitoring terminal in UAV aerial photography detection tasks, this embodiment constructs a host computer system based on the Qt for Python framework. This system effectively solves technical problems in existing technologies such as interface response delay, chaotic display of high-density targets, and unknown edge state through an asynchronous multi-threaded architecture, semantic-level post-processing filtering, and a UDP telemetry mechanism. This system designs an asynchronous multi-threaded architecture with a producer-consumer model. The main thread is only responsible for interface rendering and user interaction; the child threads, namely DetectionThread and RTSPReceiveThread in the code, are responsible for time-consuming video stream decoding and deep learning inference. The two communicate through Qt signals and slots. When a child thread completes the inference of a frame, it sends a result_ready signal carrying an image data pointer and detection result. The main thread's slot function responds to this signal and triggers a redraw. Regardless of the background inference time, the main interface always maintains a smooth refresh rate of 60 FPS, and the button click response time is <50ms, achieving a "zero-lag" real-time monitoring experience, demonstrating the significant advantages of the asynchronous decoupled architecture in ensuring real-time performance.

[0122] Drone aerial photography scenarios are characterized by extremely dense targets and a wide variety of categories. Existing systems typically render all detection boxes directly, resulting in the image being covered by hundreds of overlapping bounding boxes, obscuring key targets and preventing operators from effectively obtaining crucial information. This invention constructs a category filtering module. This module inserts a logical mask filter between the model inference output layer and the interface rendering layer. The system pre-sets category whitelists for scenarios such as "personnel reconnaissance" and "vehicle statistics." During the rendering phase, the system iterates through the detection result tensor, performs a logical AND operation between the target's class_id and the whitelist, and only calls the drawing engine to draw when the result is true. This mechanism significantly improves the operator's visual search efficiency in complex scenes and effectively solves the information overload problem under large-scale aerial photography data.

[0123] Existing UAV-based airborne monitoring systems typically operate as "black boxes," meaning ground operators have no access to the real-time temperature and load of the onboard computing chip. In high-temperature summer environments, the equipment is prone to overheating and frequency throttling, leading to monitoring failures without prior warning from the operator. This invention implements a UDP telemetry monitoring mechanism. A daemon process runs at the edge, reading NPU / CPU temperature and utilization through system file nodes, packaging it into JSON format, and broadcasting it via UDP. The SystemMonitorThread thread of the QT system listens on a designated port, parses the JSON data, and drives the dashboard component. The system incorporates threshold comparison logic; when the temperature is too high, it triggers a dynamic switch in the interface stylesheet, rendering the dashboard color as a red warning.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting small targets in UAV aerial photography based on multi-path fusion perception, characterized in that, The steps include the following: Step 1: Using YOLOv11s as the baseline model, a dual-coordinate space semantic fusion module is introduced, and a lightweight detection model framework is built by combining multi-path fusion and attention enhancement mechanisms; the model is trained and optimized on the drone aerial photography small target dataset, and the model iteration of multi-dimensional feature fusion is completed to obtain a trained detection model that can be used for inference. Step 2: Based on the detection model obtained in Step 1, the model is deployed and optimized for inference on the Xiangchengpai AI PRO edge computing unit through four steps: format conversion and graph optimization, mixed precision quantization and compilation, inference engine construction, post-processing and result mapping. Real-time inference calculations are performed on drone aerial images / video streams to obtain accurate detection results including target category, location coordinates and confidence level. Step 3: Based on the accurate edge detection results obtained in Step 2, synchronously collect the operating status data of the airborne hardware equipment, complete the two-way data interaction between the airborne terminal and the ground station through the communication transmission module, and then realize the rendering and display of the detection results, category filtering, multi-channel monitoring and real-time feedback of equipment status by the ground station's visual monitoring terminal. Combined with the human-machine interaction interface, remote control of the detection process is realized, forming a closed loop of drone aerial photography small target detection and monitoring.

2. The method for detecting small targets in UAV aerial photography based on multi-path fusion perception according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: First, build the detection model architecture and input the feature map into the feature enhancement module to improve the feature representation of small targets; Step 1.2 involves performing channel attention and spatial attention processing through dual-channel feature separation, combining the network layers to complete multi-path feature fusion, and using a drone aerial photography small target dataset to complete model training, validation, and optimization, ultimately obtaining a trained detection model that can be used for inference.

3. The method for detecting small targets in UAV aerial photography based on multi-path fusion perception according to claim 1, characterized in that, Step 2 is as follows: Step 2.1, Model Format Conversion and Graph Optimization: Export the detection model obtained in Step 1 to ONNX format, fuse convolution, batch normalization, and activation function layers, remove redundant nodes during the training phase, and standardize the custom operators of the dual-coordinate space semantic fusion module. Step 2.2, Hybrid Precision Quantization and Compilation: Convert the ONNX format model into an inference file adapted to Orange Pi, select typical aerial images of small targets for calibration, use INT8 quantization for the backbone network, standardize the dual coordinate space semantic fusion module and the detection head to retain FP16 precision, and compile to generate a heterogeneous computation graph adapted to the Orange Pi architecture. Step 2.3, Edge Inference Engine Construction: Based on the C++ / Python engine, the underlying API is called to load the compiled model. Zero-copy memory allocation is used to reduce data copy latency. RGA hardware / CPU multi-threading is used to scale, normalize and perform other preprocessing on the drone aerial video stream before inputting it into the model. Step 2.4, Post-processing and Result Mapping: The NPU performs parallel inference on the pre-processed aerial data, outputting target category probability and position offset tensors. Then, the CPU executes the non-maximum suppression algorithm to decode the relative offset into the absolute coordinates of the original image, filtering out low-confidence and overlapping redundant candidate boxes, and finally obtaining accurate small target detection results at the edge.

4. The method for detecting small targets in UAV aerial photography based on multi-path fusion perception according to claim 3, characterized in that, Step 3 specifically involves: Step 3.1, Data Acquisition and Encoding Transmission: The airborne terminal synchronously acquires the detection results from Step 2, as well as NPU utilization, CPU temperature, and other device status data. After compressing and encoding the video stream and data, the detection results are transmitted through the Socket interface, and the device status data packets are broadcast to the ground station via the UDP protocol. Step 3.2, Ground station data reception and processing: The ground station visual monitoring terminal receives and decodes all data transmitted from the airborne terminal. The multi-source data management module completes the data source adaptation, and the asynchronous inference engine completes image decoding and data post-processing through an independent working thread. The processing results are asynchronously transmitted to the main interface thread through the signal and slot mechanism. Step 3.3, Visualization and Intelligent Filtering of Detection Results: The rendering layer overlays the target bounding box, category label and confidence score on the original aerial video stream. The semantic category filtering module performs logical filtering on the detection results of Step 2 according to user needs, only displaying the filtered targets and statistically analyzing the category distribution in real time. Step 3.4, Hardware Status Monitoring and Human-Machine Interaction: The ground station analyzes the status data of the airborne equipment, visualizes the hardware operation status through the instrument panel, completes the abnormal warning through the built-in threshold comparison logic, and provides human-machine interaction interfaces such as confidence threshold adjustment and detection mode switching to realize remote control of the detection process and form a complete detection and monitoring closed loop.

5. A UAV aerial photography small target detection system based on multi-path fusion perception, used to apply the UAV aerial photography small target detection method based on multi-path fusion perception as described in any one of claims 1-4, characterized in that, include: The airborne edge computing subsystem, mounted on an unmanned aerial vehicle (UAV) platform, integrates all the hardware and software resources of the edge inference subsystem. It provides physical support and operating environment for image acquisition, model inference, and status acquisition at the edge. At the same time, it serves as the airborne data source for the communication transmission module, enabling the unified collection and external transmission of detection data and equipment status data. The communication transmission module connects the airborne edge computing subsystem and edge inference subsystem with the ground station's visual monitoring terminal, enabling two-way data interaction. It is responsible for the real-time encoding, compression, and transmission of aerial video streams, target detection results, and airborne equipment status data. At the same time, it transmits human-machine interaction control commands from the ground station back to the airborne terminal, ensuring the real-time performance and integrity of data transmission and the accurate delivery of commands. The edge inference subsystem, deployed on the Orange Pie AI PRO hardware platform of the airborne edge computing subsystem, is responsible for the acquisition, preprocessing, small target detection model inference, and post-processing of detection results of drone aerial images / video streams. At the same time, it collects real-time operating status data of the airborne hardware and is the core functional unit for realizing real-time small target detection at the edge. The visual monitoring terminal, deployed at the ground station, is developed using Python based on a client-server architecture and a cross-platform graphical interface framework. It is responsible for receiving and decoding all data transmitted by the communication transmission module, realizing the visualization rendering of detection results, multi-source data management, intelligent category filtering, and multi-channel parallel monitoring. At the same time, it monitors the status of airborne hardware in real time and provides anomaly warnings. It provides a visual human-machine interaction control interface, allowing ground operators to intuitively grasp the detection process and results.

6. The UAV aerial photography small target detection system based on multi-path fusion perception according to claim 5, characterized in that, The edge reasoning subsystem includes: The image acquisition unit acquires high-resolution, real-time drone aerial video streams, providing raw input data for model inference and serving as the data source entry point for edge inference. The NPU inference acceleration unit loads a lightweight detection model compiled with mixed precision quantization, completes parallel inference operations on the aerial video stream, and outputs the raw tensor data of the target detection. It is the core computing unit for edge inference. The telemetry data broadcasting unit collects real-time operational status data of the airborne hardware, packages it according to specifications, and outputs it to the communication transmission module, providing data support for the hardware status monitoring of the ground station's visual monitoring terminal.

7. The UAV aerial photography small target detection system based on multi-path fusion perception according to claim 5, characterized in that, The visual monitoring terminal includes: The multi-source data management module enables flexible switching and unified management of system input data sources, adapts to various data input scenarios of drone aerial photography, and provides a stable and standardized data source for subsequent inference post-processing and visualization. The asynchronous inference engine separates interface rendering from background data processing tasks, avoiding time-consuming decoding and post-processing operations that cause terminal interface lag, and ensuring the real-time performance and smoothness of visual monitoring. The rendering layer module intuitively overlays the accurate detection results output by the edge inference subsystem onto the original aerial video stream, realizing a visual display of the detection results, allowing operators to clearly understand the location, category, and confidence level of the target; The semantic category filtering module solves the problem of information overload caused by dense targets and numerous categories in drone aerial photography scenarios, enabling accurate filtering of detection results and improving the operator's visual search efficiency; The multi-channel parallel monitoring module supports simultaneous monitoring of multiple aerial video streams, adapting to application scenarios involving multiple drone cameras or collaborative detection by multiple drones, and enabling parallel visualization of multiple video streams and corresponding detection results.

8. The UAV aerial photography small target detection system based on multi-path fusion perception according to claim 7, characterized in that, The semantic category filtering module includes: The preset scheme management unit stores a category whitelist for specific application scenarios. The preset scenarios include personnel reconnaissance, vehicle statistics, and traffic flow monitoring. Operators can directly select the appropriate preset scheme. The post-processing filtering unit receives all detection box data output by the edge inference subsystem, and performs logical AND filtering operations on the detection results based on the user's currently active category whitelist or keywords entered in the search box to remove the target data that did not match. The dynamic rendering unit only visualizes and draws the targets filtered by the post-processing filtering unit, while simultaneously calculating the number and category distribution of the filtered targets in real time and displaying the statistical results intuitively on the terminal interface.

Images