A method and system for UAV target detection based on visual detection algorithms

Abstract

This application discloses a UAV target detection method and system based on a visual detection algorithm. It generates UAV cruise route data containing GPS coordinate sequences through a path planning algorithm; acquires multiple frames of RGB image data of streetlights using a visible light camera; and simultaneously acquires single-channel infrared thermal radiation image data of corresponding spatiotemporal locations using an onboard infrared thermal imaging module. A mapping relationship is established between the RGB image data and the infrared thermal radiation image data to form a multimodal raw dataset D. The multimodal raw dataset D is preprocessed to obtain a standardized multimodal dataset. Based on the standardized multimodal dataset, an improved UAV-YOLO model is constructed to identify streetlight heat sources. Addressing the issue that streetlight heat sources appear as small hot spots (10-30 pixels) in images from the UAV's top-down perspective, and are often mixed with other heat sources in the urban environment (cars, air conditioner outdoor units, etc.), traditional algorithms have low accuracy in distinguishing them. This application improves detection accuracy.

Description

Technical Field

[0001] This application relates to the field of street light detection, and in particular to a method and system for UAV target detection based on a visual detection algorithm. Background Technology

[0002] With the advancement of smart city construction, the demand for intelligent operation and maintenance management of urban street light systems, as important infrastructure, is becoming increasingly urgent. Traditional manual inspection methods suffer from problems such as low efficiency, high cost, and incomplete coverage. Drone inspection technology, with its advantages of high mobility, wide coverage, and high degree of automation, is gradually becoming an important technical means for street light operation and maintenance.

[0003] However, existing drone-based street light inspection technology mainly relies on visible light cameras for fault detection, which has the following technical shortcomings in practical applications:

[0004] (1) The problem of blind spots in the top-down angle detection: The light-emitting angle of street lights is generally 120°-150° towards the ground. When the UAV is inspecting from a height of 30-80 meters, the visible light camera has difficulty directly observing the light source status facing the ground. This structural blind spot means that even if the street light is emitting light normally, it may be misjudged as a fault. According to statistics, the misjudgment rate is as high as 32.7%.

[0005] (2) Small target heat sources are difficult to identify. From the top-down angle of the UAV, the heat source of the street lamp appears as a small hot spot of only 10-30 pixels in the infrared image. At the same time, there are a lot of interfering heat sources in the urban environment, such as car engines (temperature 60-80℃), air conditioner outdoor units (temperature 35-50℃), and heat reflection from buildings. These heat sources are highly similar to the hot spot of the street lamp in size and temperature range. Traditional image processing algorithms and deep learning models have difficulty in accurately distinguishing them, resulting in a street lamp heat source identification accuracy of only 68.5%.

[0006] (3) Lack of multimodal feature fusion: Existing visual detection models are mainly designed for RGB three-channel images and do not make full use of the temperature distribution information of infrared thermal imaging. Although infrared images can directly reflect the distribution of heat sources, their low contrast and high noise characteristics limit the detection accuracy. The lack of an effective RGB-infrared multimodal fusion mechanism makes it impossible to give full play to the advantages of the structural information of visible light images and the temperature information of infrared images.

[0007] (4) Edge computing resources are limited. Existing deep learning models (such as the YOLO series) use FP32 precision and have a large model size (about 10.2MB). On the ARM architecture edge computing platform (such as RK3399) carried by the drone, the inference speed is only 15-18FPS, which cannot meet the real-time detection requirement of 30FPS. If GPU acceleration is used, the power consumption will rise to more than 8.5W, which will shorten the drone's flight time from 2.5 hours to 1.2 hours, severely limiting the inspection coverage.

[0008] Therefore, there is an urgent need for a lightweight detection method that can accurately identify heat sources of small streetlights, effectively integrate multimodal features, and is suitable for edge deployment. Summary of the Invention

[0009] In the case of streetlights appearing as small hot spots (10-30 pixels) in images from the top-down angle of a drone, and being mixed with other heat sources in the urban environment (cars, air conditioner outdoor units, etc.), traditional algorithms have low accuracy in distinguishing them. This application provides a drone target detection method and system based on a visual detection algorithm to improve detection accuracy.

[0010] One aspect of this application provides a UAV target detection method based on a visual detection algorithm, comprising: S1, generating UAV cruise route data containing GPS coordinate sequences based on preset street light geographic information system map data using a path planning algorithm; S2, controlling the UAV to fly based on the cruise route data, acquiring multiple frames of street light RGB image data using a visible light camera for extracting street light structural features; simultaneously acquiring single-channel infrared thermal radiation image data of corresponding spatiotemporal locations using an onboard infrared thermal imaging module for heat source detection; establishing a mapping relationship between the RGB image data and the infrared thermal radiation image data to form a multimodal raw dataset D; S3, preprocessing the multimodal raw dataset D to obtain a standardized multimodal dataset. S4, based on a standardized multimodal dataset S5. Construct an improved UAV-YOLO model to identify street light heat sources; S6. Quantize the improved UAV-YOLO model with INT8 and prune channels to obtain a lightweight model, and deploy the lightweight model to the UAV computing platform.

[0011] Furthermore, S3, obtains the standardized multimodal dataset. Including: infrared thermal radiation images Adaptive denoising based on connected component analysis is performed to obtain the denoised image. For denoised images The enhanced image is obtained by segmenting the temperature histogram and enhancing local contrast. The Hough transform is used to extract the vertical line segment features of the streetlight poles from the RGB image data. The calibration matrix H is then used to map the coordinates of the vertical line segment features of the streetlight poles to the infrared image coordinate system, thus obtaining the streetlight location map. According to the street light location map For image enhancement Differential bilinear interpolation scaling and piecewise temperature normalization are performed to obtain normalized infrared images. Temperature mapping parameters Based on the connected component analysis results and inter-frame optical flow calculation results, from the normalized infrared image The morphological feature vector F and motion feature vector V of the hot spot are extracted to obtain the standardized multimodal dataset. The morphological feature vector F includes the area, circularity, aspect ratio, and centroid coordinates of the hot spot, while the motion feature vector V includes the displacement and direction of motion of the hot spot between adjacent frames. This represents a normalized RGB image; GPS represents GPS coordinates, and t represents a timestamp.

[0012] Furthermore, an enhanced image is obtained. Including: statistically denoised images Temperature value distribution, generate temperature histogram ; temperature histogram The image is divided into multiple temperature zones, each corresponding to a heat source from an urban facility. The pixel spatial distribution density of each temperature zone is calculated, defined as the ratio of the number of pixels within that temperature zone to the total number of pixels in the image. Based on the pixel spatial distribution density of each temperature zone, corresponding contrast limiting parameters are set. The image is then divided into multiple local blocks, and the dominant temperature zone for each block is determined. Adaptive histogram equalization is performed using the contrast limiting parameters corresponding to the dominant temperature zone to limit contrast, outputting an enhanced image with improved local contrast. .

[0013] Specifically, this application first uses temperature histograms. Statistical analysis divides the continuous temperature distribution into discrete temperature intervals, each corresponding to a specific type of urban heat source. This division is based on the physical heating characteristics of different heat sources: the operating temperature of LED streetlights typically exhibits a Gaussian distribution within the range of 40-55℃; the temperature of car engine heat sources is within the range of 60-80℃ and is relatively dispersed; the temperature of air conditioner outdoor units is between 35-50℃ but exhibits periodic variations. By dividing the temperature intervals, the characteristics of heat sources in the physical world are mapped to the image processing domain, providing a theoretical basis for subsequent differential enhancement.

[0014] The pixel spatial distribution density metric cleverly utilizes the fundamental differences in spatial distribution between streetlight heat sources and other urban heat sources. Streetlight heat sources appear as compact, circular hot spots in top-view images, with highly concentrated pixels within their temperature range, resulting in a high spatial distribution density. In contrast, heat sources such as cars and air conditioner outdoor units, due to their irregular shapes or dispersed locations, exhibit relatively low pixel spatial distribution density. This density difference provides an important criterion for distinguishing between different types of heat sources.

[0015] Traditional global contrast enhancement methods amplify the contrast across all temperature ranges simultaneously, leading to excessive amplification of background noise and consequently reducing the signal-to-noise ratio of target detection. This application dynamically sets contrast limiting parameters based on the pixel spatial distribution density of each temperature range, achieving refined enhancement control: for the high-density temperature range where the streetlight heat source is located, a larger contrast limiting parameter is used to fully enhance its detailed features; for the low-density temperature range of the background heat source, a smaller contrast limiting parameter is used to suppress noise amplification.

[0016] The image is divided into multiple local blocks, and the dominant temperature range of each block is determined. In UAV overhead images, heat sources at different locations may be in different temperature ranges. A globally uniform enhancement strategy may lead to over-enhancement or under-enhancement of some areas. Through adaptive processing of local blocks, each block adopts the optimal contrast parameters according to its dominant temperature range, which ensures sufficient enhancement of the street lamp hotspots while avoiding over-enhancement of other heat source areas, thus achieving spatial adaptability of image enhancement.

[0017] This application combines the classic CLAHE (Contrast Limiting Adaptive Histogram Equalization) algorithm with temperature distribution characteristics to form an adaptive enhancement mechanism for temperature sensing. Unlike traditional CLAHE, which uses fixed contrast limiting parameters, this application dynamically adjusts the parameters according to the temperature range and spatial distribution density. This makes the enhancement process more closely match the physical characteristics of different heat sources in the infrared image, thereby enhancing the visibility of small streetlight hotspots while maintaining the distinguishability from heat sources in other cities.

[0018] Furthermore, a normalized infrared image is obtained. Temperature mapping parameters This includes: based on the street light location map The location of streetlights in the image is used to determine the neighborhood and background regions of the streetlights; this is used to enhance the image. The image is scaled to a preset target size using bilinear interpolation, with dense sampling in the neighborhood of streetlights and sparse sampling in the background. The temperature range of the scaled image is then extracted. A piecewise linear temperature mapping function is constructed based on the heat source characteristics of different temperature ranges. Using piecewise linear temperature mapping functions The scaled image is normalized pixel by pixel to obtain a normalized infrared image. ;

[0019] In particular, this application utilizes a street light location map. As prior knowledge, the image space is divided into the neighborhood region of the streetlights and the background region. In the UAV top-down image, although the hot spots of the streetlights are small in size (10-30 pixels), their positions are predictable. By extracting the streetlight pole position from the RGB image, potential hot spot regions can be accurately located. The introduction of this positional prior allows subsequent processing to allocate computational resources selectively, avoiding the inefficiency of uniformly processing the entire image in traditional methods.

[0020] Traditional image scaling employs a uniform sampling strategy, which can easily lead to the loss of crucial information for small targets. Therefore, this application proposes bilinear interpolation:

[0021] Dense sampling in the neighborhood of streetlights: For streetlight hotspots of 10-30 pixels, dense sampling ensures that sufficient temperature gradient information is preserved during interpolation. From a signal processing perspective, this is equivalent to increasing the sampling frequency in the region of interest, satisfying the Nyquist sampling theorem's requirement for small-sized high-frequency signals, and preventing hotspot feature degradation caused by aliasing.

[0022] Sparse sampling in the background region: For large heat sources such as cars and air conditioner outdoor units in the background region, sparse sampling can preserve their overall temperature distribution characteristics while reducing computational complexity. This strategy is based on the matching principle between heat source size and sampling density; the low-frequency characteristics of large heat sources can be accurately characterized by sparse sampling.

[0023] Furthermore, in S4, a model based on the improved UAV-YOLO is constructed, including: standardizing the multimodal dataset. Normalized infrared images Normalized RGB image Streetlight location map As input, a three-channel input tensor is constructed; an improved UAV-YOLO model based on the YOLOv8 architecture is constructed, replacing the original C2f module with a C2f-BRA module in the YOLOv8 backbone network; the C2f-BRA module extracts features from the input three-channel tensor, divides the extracted feature map into multiple sub-regions through region partitioning, calculates the similarity matrix between sub-regions, selects the top-K sub-regions with the highest similarity for sparse attention calculation, and outputs an enhanced feature map; in the neck network of YOLOv8, an adaptive feature pyramid network AFPN is used to replace... The original path aggregation network (PAN) receives multi-scale enhanced feature maps from the backbone network. Through a bottom-up, layer-by-layer fusion strategy and an adaptive spatial weighting mechanism, it fuses features from different scales to generate a fused multi-scale feature representation. The morphological feature vector F and motion feature vector V are input into a feature encoder composed of fully connected layers and concatenated with the multi-scale features output by AFPN along the channel dimension to obtain enhanced features. These enhanced features are then input into the YOLOv8 detector head, which outputs the location coordinates, confidence scores, and class predictions of heat source candidate boxes, and determines the classification based on temperature mapping parameters. The normalized temperature values ​​within the candidate boxes are inversely mapped to obtain heat source detection results that include the actual temperature values; a standardized multimodal dataset is used. Using GPS coordinates and timestamp t, spatiotemporal correlation analysis is performed on the heat source detection results to generate a spatiotemporal distribution map of streetlight heat sources.

[0024] In particular, traditional target detection models only process single-modal data and cannot fully utilize the complementary information from different sensors. This application constructs a three-channel input tensor:

[0025] Infrared channel: Provides temperature distribution information of heat source, directly reflecting the heating characteristics of street lamp, but lacks structural details; RGB channel: Provides geometric structure and texture information of street lamp, which helps to locate the street lamp position, but cannot reflect the thermal state; Location map channel: Provides spatial prior information of street lamp, guiding the model to focus on potential heat source areas.

[0026] The C2f-BRA (Block Region Attention) module is designed for small target detection. Streetlight heat spots exhibit localized clustering characteristics in images, while traditional global attention mechanisms are diluted by large areas of background information. By calculating the similarity matrix between sub-regions, it can identify groups of regions containing similar heat source features. The top-K sub-regions with the highest similarity are selected for attention calculation, achieving precise allocation of computational resources. For streetlight heat spots, this mechanism can automatically focus on small regions with abnormal temperatures, suppressing interference from background heat sources. Theoretically, when K is set to 10%–15% of the total number of regions in the image, it can cover all potential streetlight heat source regions while reducing the computational complexity from O(n²) to O(nk). The sparse attention mechanism dynamically allocates weights based on the feature similarity between regions, giving regions with streetlight heat source features higher attention weights and enhancing the feature representation ability of small targets.

[0027] The Adaptive Feature Pyramid Network (AFPN) addresses the information loss problem of small targets in multi-scale features: unlike the simple feature addition of traditional FPN, AFPN preserves unique information at each scale through progressive fusion. For streetlight hotspots of 10-30 pixels, low-level features retain precise location information, while high-level features provide semantic discriminative capabilities. Progressive fusion ensures the effective combination of these two types of information. This mechanism automatically adjusts the fusion weights based on the contribution of features at different scales. Experiments show that for small hotspot detection, the weight of low-level features (1 / 8 scale) is approximately 0.6, the weight of mid-level features (1 / 16 scale) is approximately 0.3, and the weight of high-level features (1 / 32 scale) is approximately 0.1. This weight allocation maximizes the detection performance of small targets.

[0028] Furthermore, streetlight heat spots possess unique morphological characteristics—approximately circular, with a stable area (100-900 pixels) and high circularity (>0.8). These characteristics contrast sharply with the linear heat sources of car exhaust pipes and the rectangular heat sources of air conditioner outdoor units. Therefore, this application introduces a morphological feature vector F to provide a strong discriminative basis for target classification. Streetlights, as static heat sources, have near-zero inter-frame displacement, while car heat sources exhibit obvious motion trajectories. The motion feature vector V obtained through optical flow calculation can effectively filter out dynamic heat source interference, simplifying the detection problem from multi-class heat source classification to static heat source detection.

[0029] Furthermore, in S5, channel pruning is performed on the improved UAV-YOLO model, including: during the training of the improved UAV-YOLO model, identifying all batch normalization layers in the network and applying L1 regularization constraints to the weight parameters of each batch normalization layer; after training, extracting the weight parameters of all batch normalization layers, calculating the absolute value of each weight and sorting them, and determining the pruning threshold according to a preset pruning rate; for channels whose absolute weight value is less than the pruning threshold, creating a binary mask to mark the corresponding channels as channels to be pruned, and applying the mask to remove the corresponding convolutional kernels and batch normalization parameters; using a standardized multimodal dataset. Fine-tuning and training were performed on the pruned UAV-YOLO model.

[0030] In particular, the L1 norm, as a sparsity-induced norm in convex optimization, has a gradient discontinuity at zero that drives the optimization process to push unimportant weights towards zero. This is relevant for the scaling factor of batch normalization layers. L1 regularization enables the network to automatically learn the importance of each channel during training, with important channels... The value remains relatively large, while the redundant channels... The value approaches zero. In the streetlight heat source detection task, L1 regularization preserves channels that are sensitive to small-sized hot spot features, such as edge detection channels and temperature gradient channels, while suppressing redundant channels that respond to background textures. This task-driven channel selection is better at preserving the model's ability to detect small targets than random pruning.

[0031] Furthermore, by uniformly ranking the weights of all layers, information bottlenecks caused by excessive pruning of certain layers are avoided. For tasks involving the detection of small targets of 10-30 pixels, the detail feature channels of shallow networks are particularly important, and global ranking ensures that these key channels are preserved. Different layers have different channel redundancy, and uniform ranking allows the network to adaptively determine the pruning depth of each layer. Experiments show that the pruning rate of deep networks (semantic features) can reach 60%, while the pruning rate of shallow networks (detail features) is only 20%, a distribution that meets the feature requirements of small target detection.

[0032] This structured channel pruning method fundamentally solves the contradiction between high model complexity and limited hardware resources faced by deep models when deployed on edge devices. Through intelligent channel selection and optimization strategies, it achieves efficient model compression while maintaining small target detection capabilities.

[0033] Furthermore, the improved UAV-YOLO model undergoes INT8 quantization, including: forward inference of the fine-tuned pruned UAV-YOLO model using the standardized multimodal dataset D'; during forward inference, the activation values ​​output by each convolutional layer, batch normalization layer, and activation function layer in the pruned UAV-YOLO model are obtained, and the numerical distribution range and histogram of the activation values ​​of each layer are statistically analyzed; based on the statistical distribution of the activation values ​​of each layer, the optimal quantization scale factor and zero offset of each layer are calculated using the quantization error minimization criterion, where the optimal quantization scale factor maps the floating-point numerical range to 8 bits. Integer range; using the calculated optimal quantization scaling factor and zero-point offset, the 32-bit floating-point weight parameters and activation values ​​in the pruned UAV-YOLO model are quantized into 8-bit integer representations to generate an INT8 quantized model; the INT8 quantized model is exported to ONNX format, and the ONNX model is optimized for inference graph and operator fusion using the TensorRT optimization engine to generate a TensorRT inference engine file adapted to the UAV computing platform; the TensorRT inference engine file is deployed to the edge computing device carried by the UAV to realize real-time heat source detection of the collected multimodal data.

[0034] Specifically, streetlight hotspots appear as sparse high-activation points (corresponding to temperature anomaly areas) in shallow networks, while they appear as compact feature clusters in deep networks. By analyzing the activation value histogram, it was found that the activation values ​​of the key layers for detecting small hotspots exhibit a long-tailed distribution, meaning that most values ​​are concentrated in the low range, while a few key values ​​are located in the high range.

[0035] For key layers (such as convolutional layers 2-4) detecting hotspots of 10-30 pixels, the dynamic range of activation values ​​can reach [-50, 200], while the range of background feature layers is only [-10, 30]. An adaptive strategy sets different quantization parameters for different layers to ensure that the weak features of small targets are not overwhelmed by quantization noise.

[0036] Furthermore, analysis revealed that while the activation values ​​of neurons detecting hotspots of 10-30 pixels were sparse, their amplitudes were relatively large. The adopted quantization strategy automatically assigned more quantization levels to these outlier activation values, preventing small target features from degenerating into background noise during the quantization process. INT8 quantization compressed gradient information by a factor of 16 (from FP32 to INT8), but through an optimized scale factor, the fidelity of temperature gradient information was ensured.

[0037] This INT8 quantization method for small target protection fundamentally solves the contradiction between high accuracy requirements and limited computing resources faced by deep learning models when deployed at the edge. Through the deep integration of fine quantization parameter optimization and hardware acceleration technology, it achieves an order-of-magnitude improvement in inference performance with almost no loss of small target detection capability.

[0038] Furthermore, establishing a mapping relationship between RGB image data and infrared thermal radiation image data includes: jointly calibrating the visible light camera and the infrared thermal imaging module to obtain the intrinsic parameter matrix, distortion coefficient, and extrinsic parameter matrix of the relative positions of the two imaging devices, and constructing a calibration matrix H; during data acquisition, hardware triggering is used to achieve time synchronization between the visible light camera and the infrared thermal imaging module so that the acquired RGB image data and single-channel infrared thermal radiation image data have the same timestamp; using the calibration matrix H and the extrinsic parameter matrix, the pixels in the RGB image coordinate system are mapped to the infrared thermal radiation image coordinate system; for each frame of data acquired, the corresponding GPS coordinate position, timestamp t, UAV flight altitude, and attitude data are recorded; and RGB images with the same timestamp are mapped... Infrared thermal radiation images The data, including GPS coordinates, timestamps (t), and flight parameters, are associated and stored to construct data tuples. Based on all collected data tuples, a multimodal raw dataset is constructed. Each data tuple contains a spatiotemporally aligned RGB image, an infrared image, its corresponding geographic location information, and a calibration matrix H.

[0039] Furthermore, S1, a path planning algorithm is used to generate drone cruise route data containing GPS coordinate sequences, including: reading street light distribution data from the smart city street light geographic information system, including the unique identifier ID, latitude and longitude coordinates, installation height, street light model, and power data of each street light; constructing a spatial distribution network map of street lights based on their latitude and longitude coordinates, where street lights are nodes, roads between adjacent street lights are edges, and the weight of the edges is set to the Euclidean distance between street lights; setting drone cruise parameters, including cruise altitude, cruise speed, and the acquisition frame rate of the infrared thermal imaging module; and then, based on the cruise... The ground coverage of a single frame image is calculated using the altitude and field of view of the infrared thermal imaging module. Based on the cruise speed and acquisition frame rate, the spatial interval between adjacent acquisition points is calculated. The A* path planning algorithm is used to search for the optimal cruise path on the street light spatial distribution network map, with the shortest total flight distance as the optimization objective, enabling the UAV to traverse all target street lights. Image acquisition waypoints are inserted along the generated optimal path according to the calculated spatial interval; each waypoint includes longitude, latitude, and altitude. The discrete waypoint sequence is smoothed to generate a continuous flight trajectory, resulting in the UAV cruise route data.

[0040] Another aspect of this application provides a drone street light heat source detection system based on adaptive sparse attention, for performing a drone target detection method based on a vision detection algorithm.

[0041] Compared to existing technologies, the advantages of this application are:

[0042] To address the problem in existing technologies where streetlight heat sources appear as small hot spots of 10-30 pixels in infrared images captured by drones from above, highly mixed with other heat sources in the urban environment such as car engines, air conditioner outdoor units, and building reflections, traditional target detection algorithms struggle to accurately distinguish streetlight heat sources from background interference, leading to high false positive and false negative rates. This application provides a drone target detection method based on a vision detection algorithm. This method achieves the following: by fusing the streetlight structural features from RGB images and the temperature distribution features from infrared images, it adaptively focuses on the small hot spot region at the streetlight location using a C2f-BRA sparse attention mechanism, suppressing interference from non-streetlight heat sources; by extracting the morphological and motion features of the hot spot and combining prior information about the streetlight location, it accurately distinguishes streetlight heat sources from other urban heat sources; by using AFPN multi-scale feature fusion to enhance small target detection capabilities and improve the accuracy of streetlight heat source detection, it also enables real-time deployment on the drone platform through model lightweighting technology. Attached Figure Description

[0043] This application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0044] Figure 1 This is an exemplary flowchart of a UAV target detection method based on a visual detection algorithm, according to some embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the C2f-BRA module according to some embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the AFPN feature pyramid structure according to some embodiments of this application. Detailed Implementation

[0047] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0048] Example 1

[0049] like Figure 1As shown, based on the preset street light geographic information system map data, a path planning algorithm generates drone cruise route data containing GPS coordinate sequences. Based on the cruise route data, the drone is controlled to fly, acquiring multiple frames of RGB image data of the street lights using a visible light camera to extract street light structural features. Simultaneously, an onboard infrared thermal imaging module acquires single-channel infrared thermal radiation image data of the corresponding spatiotemporal location for heat source detection. A mapping relationship is established between the RGB image data and the infrared thermal radiation image data to form a multimodal raw dataset D. The multimodal raw dataset D is preprocessed to obtain a standardized multimodal dataset. Based on standardized multimodal datasets An improved UAV-YOLO model was constructed to identify heat sources from streetlights. The improved UAV-YOLO model was quantized with INT8 and pruned to obtain a lightweight model, which was then deployed to the UAV computing platform.

[0050] Specifically, S1 generates drone cruise route data containing GPS coordinate sequences through a path planning algorithm, including: reading street light distribution data from the smart city street light geographic information system, including the unique identifier ID, latitude and longitude coordinates, installation height, street light model, and power data of each street light; constructing a spatial distribution network map of street lights based on their latitude and longitude coordinates, where street lights are nodes, roads between adjacent street lights are edges, and the edge weights are set to the Euclidean distance between street lights; setting drone cruise parameters, including cruise altitude, cruise speed, and the frame rate of the infrared thermal imaging module; and then, based on the cruise... The ground coverage of a single frame image is calculated using the altitude and field of view of the infrared thermal imaging module. Based on the cruise speed and acquisition frame rate, the spatial interval between adjacent acquisition points is calculated. The A* path planning algorithm is used to search for the optimal cruise path on the street light spatial distribution network map, with the shortest total flight distance as the optimization objective, enabling the UAV to traverse all target street lights. Image acquisition waypoints are inserted along the generated optimal path according to the calculated spatial interval; each waypoint includes longitude, latitude, and altitude. The discrete waypoint sequence is smoothed to generate a continuous flight trajectory, resulting in the UAV cruise route data.

[0051] S2, based on cruise route data, controls the drone's flight and acquires multiple frames of RGB image data of streetlights using a visible light camera to extract structural features of the streetlights. Simultaneously, it acquires single-channel infrared thermal radiation image data of the corresponding spatiotemporal location using an onboard infrared thermal imaging module for heat source detection. The visible light camera and infrared thermal imaging module are jointly calibrated to obtain the intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices of their relative positions, constructing a calibration matrix H. During data acquisition, hardware triggering enables time synchronization between the visible light camera and the infrared thermal imaging module, ensuring that the acquired RGB image data and single-channel infrared thermal radiation image data have the same timestamp.

[0052] Using the calibration matrix H and the extrinsic parameter matrix, pixels in the RGB image coordinate system are mapped to the infrared thermal radiation image coordinate system; for each frame of data acquired, the corresponding GPS coordinate position, timestamp t, UAV flight altitude, and attitude data are recorded; RGB images with the same timestamp are then mapped... Infrared thermal radiation images The data, including GPS coordinates, timestamps (t), and flight parameters, are associated and stored to construct data tuples. Based on all collected data tuples, a multimodal raw dataset is constructed. Each data tuple contains a spatiotemporally aligned RGB image, an infrared image, its corresponding geographic location information, and a calibration matrix H.

[0053] S3, preprocess the original multimodal dataset D to obtain a standardized multimodal dataset. ;

[0054] S3.1: Infrared thermal radiation image Adaptive noise reduction based on connected component analysis: hot spot regions are extracted by segmentation using a temperature threshold, and a threshold is set. Obtain a binary image ;right Perform connected component analysis and calculate the area of ​​each connected component. and roundness ,in, For the perimeter; for satisfying pixels and Connected regions are labeled as small-sized circular hotspot regions. A 3×3 median filter kernel was used; the remaining areas were marked as background areas. A 5×5 median filter kernel is used; during the filtering process, the gradient direction of the hot spot edge is calculated, and filtering is performed along the gradient normal to preserve edge features;

[0055] S3.2: Piecewise contrast enhancement based on temperature histogram statistics: Image after statistical denoising Temperature histogram Three temperature ranges were identified: [35℃, 45℃], [45℃, 65℃], and [65℃, 80℃]; these correspond to the temperature ranges of different urban facility heat sources. For each temperature range... Calculate the spatial distribution density of pixels within this interval. , where is the number of pixels in that interval; based on density Set the CLAHE parameter: When When clipLimit is set to 3.0, When clipLimit=2.0 is set, when Set clipLimit=1.5; for each 8×8 tile block, select the corresponding clipLimit parameter according to its dominant temperature range for local enhancement;

[0056] S3.3: Multimodal data alignment and normalization based on RGB feature point matching: Hough transform is used to detect vertical line segments from RGB images as candidate streetlight poles, and Hough circle transform is used to detect circular structures as candidate lampshades; the coordinates of the detected streetlight structures are then... By mapping the calibration matrix H to the infrared image coordinate system, the street light location map is obtained. During bilinear interpolation scaling, for The region within a radius of r = 20 pixels uses 4×4 neighborhood sampling, while other regions use 2×2 neighborhood sampling; a piecewise linear temperature mapping function is constructed. When T∈[45℃, 65℃], When T < 45℃, When T>65℃, ;

[0057] S3.4: Constructing a standardized dataset through feature computation and data organization: For each detected hotspot i, calculate the morphological feature vector. This includes area, circularity, aspect ratio, and centroid coordinates; the displacement vector of the hot spot between adjacent frames is calculated using the optical flow method. ,when Data smaller than 2 pixels per frame is marked as a static heat source; the processed data is organized into tuples: ,in, The normalized RGB image matrix is ​​640×640×3, with pixel values ​​ranging from [0, 1]. The normalized 640×640×1 infrared image matrix has pixel values ​​ranging from [0, 1]. It is a 640×640 binary mask matrix, with the pixel value of 1 at the potential location of the street lamp and 0 at other locations; Here, A is the morphological feature vector of the hotspot, C is the area of ​​the hotspot (in pixels), C is the circularity (dimensionless, ranging from [0, 1]), and R is the aspect ratio. The coordinates are those of the centroid. For motion feature vectors, Inter-frame displacement (unit: pixels / frame). For the magnitude of the speed of motion, The direction angle of motion (unit: radians); GPS=[longitude, latitude, altitude] represents the latitude, longitude, and altitude information of the location captured by the drone; t is the timestamp, recording the UTC time of image acquisition; For temperature mapping parameters, and Here, represents the original temperature range, and a, b, and c are the parameters of the piecewise mapping function. The above four steps are processed in parallel using CUDA streaming, with processing speed accelerated through asynchronous memory transfers and computational overlap.

[0058] S4, based on standardized multimodal datasets An improved UAV-YOLO model was constructed to identify heat sources from streetlights, including:

[0059] Standardize multimodal datasets Normalized infrared images Normalized RGB image Streetlight location map As input, construct a three-channel input tensor;

[0060] Construct an improved UAV-YOLO model based on the YOLOv8 architecture, such as... Figure 2 As shown, the original C2f module is replaced with the C2f-BRA module in the YOLOv8 backbone network. The C2f-BRA module extracts features from the input three-channel tensor, divides the extracted feature map into multiple sub-regions through region partitioning, calculates the similarity matrix between sub-regions, selects the Top-K sub-regions with the highest similarity for sparse attention calculation, and outputs the enhanced feature map.

[0061] like Figure 3 As shown, in the neck network of YOLOv8, the original path aggregation network PAN is replaced by an adaptive feature pyramid network AFPN. AFPN receives the multi-scale enhanced feature map output by the backbone network and fuses features of different scales through a bottom-up layer-by-layer fusion strategy and an adaptive spatial weighting mechanism to generate a fused multi-scale feature representation.

[0062] The morphological feature vector F and the motion feature vector V are input into the feature encoder composed of a fully connected layer, and then concatenated with the multi-scale features output by AFPN in the channel dimension to obtain the enhanced features.

[0063] The enhanced features are input into the YOLOv8 detection head, which outputs the location coordinates of the heat source candidate boxes, confidence scores, and class predictions, based on temperature mapping parameters. The normalized temperature values ​​within the candidate box are reverse-mapped to obtain heat source detection results that include the actual temperature values.

[0064] Using standardized multimodal datasets Using GPS coordinates and timestamp t, spatiotemporal correlation analysis is performed on the heat source detection results to generate a spatiotemporal distribution map of streetlight heat sources.

[0065] S5, Channel pruning is performed on the improved UAV-YOLO model, including: during the training of the improved UAV-YOLO model, identifying all batch normalization layers in the network and applying L1 regularization constraints to the weight parameters of each batch normalization layer; after training, extracting the weight parameters of all batch normalization layers, calculating the absolute value of each weight and sorting them, and determining the pruning threshold according to the preset pruning rate; for channels whose absolute weight value is less than the pruning threshold, creating a binary mask to mark the corresponding channels as channels to be pruned, and applying the mask to remove the corresponding convolutional kernels and batch normalization parameters; using a standardized multimodal dataset. Fine-tuning and training were performed on the pruned UAV-YOLO model.

[0066] The improved UAV-YOLO model was subjected to INT8 quantization, including: from a standardized multimodal dataset. A representative sample was selected to form a calibration dataset, which was then input into the fine-tuned pruned UAV-YOLO model for forward inference. During forward inference, the activation values ​​output by each convolutional layer, batch normalization layer, and activation function layer in the pruned UAV-YOLO model were collected, and the numerical distribution range and histogram of the activation values ​​of each layer were statistically analyzed. Based on the statistical distribution of the activation values ​​of each layer, the optimal quantization scale factor and zero offset of each layer were calculated using the quantization error minimization criterion. The optimal quantization scale factor was defined as mapping the floating-point numerical range to the 8-bit integer range [-128, ...]. The scaling parameters of

[127] are used; the optimal quantization scale factor and zero offset are calculated to quantize the 32-bit floating-point weight parameters and activation values ​​in the pruned UAV-YOLO model into 8-bit integer representations to generate an INT8 quantized model; the INT8 quantized model is exported to ONNX format, and the TensorRT optimization engine is used to optimize the inference graph and fuse operators in the ONNX model to generate a TensorRT inference engine file adapted to the UAV computing platform; the TensorRT inference engine file is deployed to the edge computing device carried by the UAV to realize real-time heat source detection of the collected multimodal data.

[0067] Example 2

[0068] A UAV target detection method based on a visual detection algorithm includes a UAV-borne subsystem (core execution unit) and a ground operation and maintenance subsystem (data management unit). The specific module composition and technical parameters are as follows:

[0069] Infrared thermal imaging acquisition module: mounted on a three-axis stabilized gimbal of the UAV (adjustable top angle 0-90°, control accuracy ±0.1°), used to acquire infrared thermal radiation data from streetlights; employs an uncooled microbolometer detector (NETD≤50mK, spectral response range 8-14μm, resolution 640×512, frame rate 30fps), outputting 16-bit single-channel infrared thermal imaging images (pixel values ​​are linearly mapped to temperature, temperature measurement range -20~150℃, temperature measurement accuracy ±2℃); connected to the edge chip via a USB 3.0 SuperSpeed ​​interface (transmission rate 5Gbps) to ensure frame-free image transmission;

[0070] Image preprocessing module: Integrated into the ISP unit of the RK3588 chip, used to optimize infrared image quality, containing 3 sub-units: Thermal noise suppression unit: Uses 5×5 median filtering to suppress detector salt-and-pepper noise, and eliminates isolated noise points through sliding window statistical sorting (noise suppression rate ≥92%); Contrast enhancement unit: Based on limited contrast adaptive histogram equalization (CLAHE, clipLimit=2.0, tileGridSize=8×8), enhances the grayscale difference between hot spots and the background, and improves the contrast of hot spot area by ≥40%; Size normalization unit: Uses bilinear interpolation to resize the image to 640×640 pixels (adapting to YOLOv11 input resolution), and eliminates the influence of dimensions through min-max normalization (mapping 16-bit temperature values ​​to the [0,1] interval);

[0071] Streetlight heat source detection module: Based on an improved visual algorithm architecture, deployed on the RK3588 NPU (6TOPS computing power, supporting FP16 / INT8 mixed precision computing):

[0072] The backbone network employs state-of-the-art deep separable convolutional blocks, significantly reducing the number of model parameters and computational complexity while maintaining powerful feature extraction capabilities, achieving a good balance between speed and accuracy. The network integrates a cross-stage partial connection design, enhancing gradient flow and reducing computational redundancy by splitting and fusing feature maps, enabling the network to learn rich feature representations more efficiently. Through a carefully designed hierarchical structure, the network naturally generates multi-scale feature maps at different depths.

[0073] The neck network employs a strategy combining path aggregation networks and feature pyramid networks. Through bidirectional paths—top-down and bottom-up—it fully integrates high-resolution detail information from the shallow layers of the backbone network with strong semantic information from the deeper layers. This design enhances the detection capability for targets at different scales, particularly beneficial for detecting objects of varying sizes in images, ensuring effective feature support for both large targets at close range and small targets at distant range.

[0074] The detection head employs an advanced anchor box prediction mechanism, directly predicting the center point and dimensions of the target. This simplifies the detection process, reduces reliance on prior anchor box design, and makes the model more easily generalizable to different datasets and tasks. A decoupled head design is used, separating object classification and bounding box regression tasks into different branches. This design avoids conflicts between the two tasks and has been widely proven to improve the performance of the final model, especially localization accuracy. A dynamic sample allocation strategy is used during training, intelligently separating positive and negative samples based on the degree of matching between the prediction and the ground truth.

[0075] The latitude and longitude of the faulty street light are obtained through the drone's GPS module (which supports BeiDou / GLONASS dual-mode positioning with a positioning accuracy of ≤10m) and then linked to the coordinates on the GIS map.

[0076] Model lightweighting and edge deployment module: Optimized for RK3588 hardware characteristics, containing 2 sub-units:

[0077] Model Quantization Unit: To improve the deployment efficiency of the model on edge computing devices, a hybrid quantization technique is used to optimize the model. This technique intelligently analyzes the network structure, retaining FP16 precision for specific layers with large numerical ranges or those sensitive to accuracy, while performing efficient INT8 quantization on the vast majority of other layers. While maintaining almost the original recognition accuracy of the model, the model size is compressed from 19.9MB to 11.0MB, laying the foundation for high-speed, high-precision inference on the edge NPU.

[0078] Hardware acceleration unit: It calls the operator fusion interface of RK3588 NPU (supports Conv-BN-ReLU operator fusion), enables pipeline parallelism, and decomposes the inference process into three parallel stages: feature extraction → feature fusion → detection output; at the same time, it enables low power mode (CPU frequency 1.8GHz, NPU frequency 1.0GHz, DDR frequency 2133MHz), and the power consumption of the whole machine is controlled to ≤4.8W;

[0079] Data transmission and storage module: Includes a 4G / 5G transmission unit (supports LTE Cat.12, peak rate 600Mbps) and a 128GBeMMC storage unit (supports HS400 mode, read / write speed 200MB / s); Real-time uploading of fault infrared images (JPEG format, compression ratio 10:1), GPS coordinates, fault type, and collection timestamp to the ground subsystem; Local storage of inspection data for 7 days (cyclic overlay); Support for data backtracking.

[0080] Ground maintenance subsystem: consists of an edge server (Intel Xeon Gold 6330, 32GB DDR4) and maintenance terminals (Android 13 system, supporting 4G / 5G); the server deploys a fault data management platform, supporting fault statistics (by region / fault type), GIS map annotation, and maintenance work order generation; the terminal APP supports work order reception, navigation, and maintenance result reporting, forming a closed loop of maintenance.

[0081] Example 3

[0082] This embodiment takes the inspection of 1,000 streetlights in a smart city demonstration area of ​​a certain city as the scenario (including 700 LED streetlights and 300 high-pressure sodium lamps, distributed on 5 main roads, with a spacing of 30m between each streetlight, and GIS map coordinate accuracy ≤5m).

[0083] Hardware configuration:

[0084] Drone platform: DJI Mavic 3, maximum flight time of 46 minutes, equipped with a three-axis stabilized gimbal (controllable top angle 0-90°) and Beidou / GLONASS dual-mode GPS (positioning accuracy ≤10m).

[0085] Infrared thermal imaging module: FLIR Lepton 3.5, uncooled microbolometer, resolution 640×512, NETD=50mK, frame rate 30fps, temperature measurement range -20~150℃, connected to RK3588 via USB3.0 interface;

[0086] Edge computing module: Rockchip RK3588 development board, 4-core A76 (2.4GHz) + 4-core A55 (1.8GHz) CPU, 6TOPS NPU, 4GB LPDDR4 memory, 128GB eMMC storage, supports 4G / 5G expansion;

[0087] Ground maintenance equipment: edge server (Intel Xeon Gold 6330, 32GB DDR4, 2TB SSD), maintenance terminal (Huawei MatePad Pro, Android 13 system, supports 4G network).

[0088] Software and dataset preparation:

[0089] Software environment: Ubuntu 22.04 LTS system, Linux 5.10 kernel, RKNN Toolkit 2.0 (model quantization tool), PyTorch 2.0 (model training), OpenCV 4.8.0 (image processing), TensorRT 8.6 (alternative acceleration tool);

[0090] Dataset Construction: 12,000 infrared thermal imaging images of streetlights in the demonstration area were collected (6,000 normal LED streetlights and 2,000 faulty LED streetlights; 2,000 normal high-pressure sodium lamps and 1,000 faulty high-pressure sodium lamps; 1,000 images of interfering heat sources (500 car exhaust pipes and 500 air conditioner outdoor units)). Labels were created using the LabelImg tool in COCO2017 format, with an IOU consistency pass rate of ≥95%. The dataset was divided into a training set (9,600 images) and a validation set (2,400 images) in an 8:2 ratio.

[0091] Model training: Based on the open-source algorithm framework (Ultralytics library v8.1.0), the improved model was trained on an edge server—iterated for 100 epochs, with an initial learning rate of 1e-3, a learning rate decay strategy of cosine annealing, an optimizer AdamW (β1=0.9, β2=0.999), and a weight decay of 1e-4; training was stopped when the validation set mAP@0.5 reached 98.2%, and the FP32 model (12.1MB in size) was exported.

[0092] Model Deployment: The FP32 model was quantized using RKNN Toolkit 2.0 with 100 validation images as the calibration set and a quantization error tolerance of 0.5% to generate an RKNN format model (1.8MB in size). The NPU resources were called through the RKNN API, and inference code (C++ language, based on RKNN Runtime) was written to implement the end-to-end process of image input → preprocessing → inference → result output.

[0093] Hardware connection and debugging:

[0094] Module Connections: The infrared thermal imaging module is fixed to the drone gimbal and connected to the RK3588 via a USB 3.0 cable; the 4G module is inserted into the PCIe slot of the RK3588, and the GPS module is connected via a UART interface.

[0095] Functional testing: Start the drone and test the communication of each module: The infrared module continuously collects images for 10 minutes without frame loss (stable frame rate of 30fps); the GPS module positioning error is ≤10m; the 4G module upload speed is ≥10Mbps;

[0096] Key parameter calibration:

[0097] Temperature threshold calibration: For LED streetlights (normal temperature 42-65℃) and high-pressure sodium lamps (normal temperature 55-80℃), fault thresholds of 45℃ and 58℃ were set respectively (both were taken as the lower limit of the specification plus 7%). The stability of the thresholds was verified by simulating environments of 25℃, -5℃ and 40℃ in a constant temperature chamber to ensure that there are no misjudgments within ±3℃ of temperature fluctuation.

[0098] Preprocessing parameter tuning: Test the effects of different clipLimit (1.5 / 2.0 / 2.5) and median filter kernel (3×3 / 5×5 / 7×7), and finally select clipLimit=2.0 (best hot spot contrast) and 5×5 filter kernel (92% noise suppression rate, complete detail retention).

[0099] Power consumption control: Through the power management interface of RK3588, the CPU frequency is set to 1.8GHz, the NPU frequency to 1.0GHz, and the DDR frequency to 2133MHz; the power consumption of the whole machine is monitored with a power meter and is stable at 4.8W (including the power consumption of the infrared module and 4G module).

[0100] S1: Inspection Preparation and Parameter Configuration:

[0101] S11: Route planning: Import the GIS map of the demonstration area into the UAV ground station, and use the A* algorithm to generate 5 cruise routes (each route is 6km long and covers 200 streetlights); set the cruise altitude to 50m, the speed to 8m / s, and each streetlight is covered by 3 frames of images (to ensure that the missed detection rate is ≤0.1%).

[0102] S12: Task Configuration: Create an inspection task on the ground operation and maintenance platform (Task ID: SLD-20240904-01, Target: 1000 streetlights, Person in Charge: Operation and Maintenance Group A), and synchronize the parameters to the UAV-borne system;

[0103] S2: Infrared Image Acquisition and Preprocessing:

[0104] S21: Image Acquisition: The drone takes off from the operation and maintenance center, cruises along the route, and the infrared module acquires images in real time (640×512, 16-bit). Each frame of the image is associated with GPS coordinates and timestamp.

[0105] S22: Preprocessing: Parallel processing by the RK3588's ISP unit:

[0106] ①5×5 median filtering: Eliminates detector salt-and-pepper noise, processing time 3ms;

[0107] ② Enhanced CLAHE: clipLimit=2.0, tileGridSize=8×8, time taken 4ms;

[0108] ③ Bilinear interpolation resizes to 640×640, min-max normalization, taking 2ms;

[0109] The total preprocessing time was 9ms, which meets the real-time requirements;

[0110] S31: Responsible for extracting multi-scale features composed of thermal radiation information from infrared images, with a focus on areas with significant temperature differences.

[0111] S32: Responsible for fusing infrared features at different levels of abstraction to enhance the characterization of thermal targets with varying scales (such as people and vehicles at different distances).

[0112] S33: Responsible for making the final judgment based on the fused thermal features, and outputting the category of thermal targets in the image (such as people, animals, vehicles) and their precise thermal contour bounding boxes.

[0113] The entire detection process takes ≤25ms / frame on the NPU, and the inference speed is ≥40FPS;

[0114] S4: Data Transmission and Operations Feedback

[0115] S41: Real-time upload: The 4G module uploads the infrared image (JPEG format, 150KB), GPS, and fault type of the SLD-002 to the ground server with a transmission delay of 450ms;

[0116] S42: Work order generation: The ground server marks the location of SLD-002 on the GIS map, generates a maintenance work order (work order ID: WO-20240904-001, maintenance content: replace LED driver module), and pushes it to the maintenance personnel's terminal;

[0117] S43: Closed-loop confirmation: The maintenance personnel navigate to the SLD-002 location via the terminal, upload replacement photos after the repair is completed, and the system marks the work order as closed-loop, taking 1.5 hours;

[0118] Monthly collection of misjudgment cases in the operation and maintenance closed loop (such as one case of misjudging a building air conditioner outdoor unit (temperature 48℃) as an LED street light) is added to the training set (100 new labeled images are added).

[0119] Incremental training was employed: the C2f-IRBackbone parameters were frozen, and only the heat source attention layer and detection head were fine-tuned. After 10 epochs (learning rate 1e-4), the model's heat source discrimination accuracy improved from 96.3% to 97.1%.

[0120] This embodiment completed three repeated inspections of 1000 streetlights within 3 days, covering nighttime (20:00-22:00), rainy days (rainfall of 5mm / h), and foggy / hazy days (visibility of 500m). The verification results are as follows:

[0121] Detection accuracy: 0.4% false positive rate for normal streetlights (4 cases), 0% false negative rate for faulty streetlights (all 32 faulty streetlights were identified), and 96.3% accuracy in distinguishing heat sources.

[0122] Real-time performance: Single frame processing time (acquisition + preprocessing + detection) ≤ 43ms, meeting the real-time requirement of 30 frames / second;

[0123] Battery life and power consumption: The drone inspected two routes (400 streetlights) on a single charge in 40 minutes, with 32% battery remaining and a stable power consumption of 4.8W.

[0124] Operation and maintenance efficiency: The average repair response time for 32 faulty streetlights is 1.5 hours, and the average number of streetlights repaired per person per day is 45, which is 3 times higher than the existing solution.

[0125] The foregoing illustrative description of the present application and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. The accompanying drawings are only one embodiment of the present application, and the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present application, such designs should fall within the scope of protection of this application. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for UAV target detection based on a visual detection algorithm, characterized in that, include: S1. Based on the preset street light geographic information system map data, generate drone cruise route data containing GPS coordinate sequences through a path planning algorithm; S2 controls the drone's flight based on the cruise route data and collects multiple frames of RGB image data of streetlights through a visible light camera to extract the structural features of the streetlights; Simultaneously, the built-in infrared thermal imaging module acquires single-channel infrared thermal radiation image data of the corresponding spatiotemporal location for heat source detection. Establish a mapping relationship between RGB image data and infrared thermal radiation image data to form a multimodal raw dataset D; Establishing a mapping relationship between RGB image data and infrared thermal radiation image data includes: The visible light camera and the infrared thermal imaging module are jointly calibrated to obtain the intrinsic parameter matrix, distortion coefficient, and extrinsic parameter matrix of the relative position of the two imaging devices, and a calibration matrix H is constructed. During the data acquisition process, the visible light camera and the infrared thermal imaging module are synchronized through hardware triggering so that the acquired RGB image data and single-channel infrared thermal radiation image data have the same timestamp. Using the calibration matrix H and the extrinsic parameter matrix, the pixels in the RGB image coordinate system are mapped to the infrared thermal radiation image coordinate system; For each frame of data collected, record the corresponding GPS coordinates, timestamp t, UAV flight altitude, and attitude data; RGB images with the same timestamp Infrared thermal radiation images GPS coordinates, timestamp t, and flight parameters are associated and stored to construct a data tuple; Construct a multimodal raw dataset based on all collected data tuples. Each data tuple contains a spatiotemporally aligned RGB image, an infrared image, and their corresponding geographic location information and calibration matrix H; S3, preprocess the original multimodal dataset D to obtain a standardized multimodal dataset. ; S4, based on standardized multimodal datasets An improved UAV-YOLO model was constructed to identify heat sources from streetlights. S5 quantizes the improved UAV-YOLO model using INT8 and performs channel pruning to obtain a lightweight model, which is then deployed to the UAV computing platform.

2. The UAV target detection method based on vision detection algorithm according to claim 1, characterized in that: S3, obtain the standardized multimodal dataset. ,include: Infrared thermal radiation images Adaptive denoising based on connected component analysis is performed to obtain the denoised image. ; For denoised images The enhanced image is obtained by segmenting the temperature histogram and enhancing local contrast. ; The vertical line segment features of streetlight poles are extracted from RGB image data using Hough transform, and the coordinates of these features are mapped to the infrared image coordinate system using a calibration matrix H, thus obtaining a streetlight location map. ; According to the street light location map For image enhancement Differential bilinear interpolation scaling and piecewise temperature normalization are performed to obtain normalized infrared images. Temperature mapping parameters ; Based on the connected component analysis results and inter-frame optical flow calculation results, from the normalized infrared image The morphological feature vector F and motion feature vector V of the hot spot are extracted to obtain the standardized multimodal dataset. The morphological feature vector F includes the area, circularity, aspect ratio, and centroid coordinates of the hot spot, while the motion feature vector V includes the displacement and direction of motion of the hot spot between adjacent frames.

3. The UAV target detection method based on vision detection algorithm according to claim 2, characterized in that: Get enhanced image ,include: Statistical Denoising Images Temperature value distribution, generate temperature histogram ; Temperature histogram It is divided into multiple temperature zones, and each temperature zone corresponds to a heat source for urban facilities. Calculate the pixel spatial distribution density for each temperature range. The pixel spatial distribution density is defined as the ratio of the number of pixels within a temperature range to the total number of pixels in the image. Set the corresponding contrast limit parameters based on the pixel spatial distribution density of each temperature range; The image is divided into multiple local blocks, the dominant temperature range of each block is determined, and contrast-limited adaptive histogram equalization is performed using the contrast-limiting parameters corresponding to the dominant temperature range. The result is an enhanced image with improved local contrast. .

4. The UAV target detection method based on vision detection algorithm according to claim 2, characterized in that: Obtain normalized infrared image Temperature mapping parameters ,include: According to the street light location map The location of the streetlights is used to determine the neighborhood area and background area of ​​the streetlights; Enhanced image Bilinear interpolation is used to scale the device to a preset target size, where a dense sampling strategy is used in the neighborhood of the streetlights and a sparse sampling strategy is used in the background area. Extracting the temperature range of the scaled image A piecewise linear temperature mapping function is constructed based on the heat source characteristics of different temperature ranges. ; Using piecewise linear temperature mapping function The scaled image is normalized pixel by pixel to obtain a normalized infrared image. .

5. The UAV target detection method based on a vision detection algorithm according to claim 1, characterized in that: S4, Construct a UAV-YOLO model based on the improved model, including: Standardize multimodal datasets Normalized infrared images Normalized RGB image Streetlight location map As input, construct a three-channel input tensor; An improved UAV-YOLO model based on the YOLOv8 architecture is constructed. The original C2f module is replaced with the C2f-BRA module in the YOLOv8 backbone network. The C2f-BRA module extracts features from the input three-channel tensor, divides the extracted feature map into multiple sub-regions through region partitioning, calculates the similarity matrix between sub-regions, selects the top-K sub-regions with the highest similarity for sparse attention calculation, and outputs the enhanced feature map. In the neck network of YOLOv8, the original path aggregation network PAN is replaced by an adaptive feature pyramid network AFPN. AFPN receives multi-scale enhanced feature maps output by the backbone network and fuses features of different scales through a bottom-up layer-by-layer fusion strategy and an adaptive spatial weighting mechanism to generate a fused multi-scale feature representation. The morphological feature vector F and the motion feature vector V are input into the feature encoder composed of a fully connected layer, and then concatenated with the multi-scale features output by AFPN in the channel dimension to obtain the enhanced features. The enhanced features are input into the YOLOv8 detection head, which outputs the location coordinates of the heat source candidate boxes, confidence scores, and class predictions, based on temperature mapping parameters. The normalized temperature values ​​within the candidate box are reverse-mapped to obtain heat source detection results that include the actual temperature values. Using standardized multimodal datasets Using GPS coordinates and timestamp t, spatiotemporal correlation analysis is performed on the heat source detection results to generate a spatiotemporal distribution map of streetlight heat sources.

6. The UAV target detection method based on a vision detection algorithm according to claim 5, characterized in that: S5, Channel pruning is performed on the improved UAV-YOLO model, including: During the training of the improved UAV-YOLO model, all batch normalized layers in the network are identified, and L1 regularization constraints are applied to the weight parameters of each batch normalized layer. After training, extract the weight parameters of all batch normalized layers, calculate the absolute value of each weight and sort them, and determine the pruning threshold according to the preset pruning rate. For channels whose absolute weights are less than the pruning threshold, create a binary mask to mark the corresponding channels as channels to be pruned, and apply the mask to remove the corresponding convolutional kernels and batch normalization parameters. Using standardized multimodal datasets The pruned UAV-YOLO model was fine-tuned and trained.

7. The UAV target detection method based on a vision detection algorithm according to claim 6, characterized in that: The improved UAV-YOLO model is subjected to INT8 quantization, including: Using standardized multimodal datasets Forward inference is performed on the fine-tuned trained pruned UAV-YOLO model; During the forward inference process, the activation values ​​of each convolutional layer, batch normalization layer and activation function layer in the pruned UAV-YOLO model are obtained, and the numerical distribution range and histogram of the activation values ​​of each layer are statistically analyzed. Based on the statistical distribution of activation values ​​in each layer, the optimal quantization scale factor and zero offset for each layer are calculated using the quantization error minimization criterion. The optimal quantization scale factor maps the floating-point value range to the 8-bit integer range. Using the calculated optimal quantization scale factor and zero offset, the 32-bit floating-point weight parameters and activation values ​​in the pruned UAV-YOLO model are quantized into 8-bit integer representations to generate an INT8 quantized model. The INT8 quantization model is exported to ONNX format. The TensorRT optimization engine is used to optimize the inference graph and fuse operators in the ONNX model, generating a TensorRT inference engine file adapted to the UAV computing platform. The TensorRT inference engine file is deployed to the edge computing device carried by the drone to enable real-time heat source detection of the collected multimodal data.

8. The UAV target detection method based on a vision detection algorithm according to claim 7, characterized in that: S1 generates drone cruise route data containing GPS coordinate sequences through a path planning algorithm, including: Read street light distribution data from the smart city street light geographic information system, including the unique ID of each street light, latitude and longitude coordinates, installation height, street light model and power data; A spatial distribution network diagram of streetlights is constructed based on their latitude and longitude coordinates. Streetlights are used as nodes, roads between adjacent streetlights are used as edges, and the weight of the edges is set to the Euclidean distance between streetlights. Set the drone's cruise parameters, including cruise altitude, cruise speed, and the frame rate of the infrared thermal imaging module; The ground coverage area of ​​a single frame image is calculated based on the cruising altitude and the field of view of the infrared thermal imaging module. Based on cruising speed and acquisition frame rate, calculate the spatial interval between adjacent acquisition points; The A* path planning algorithm is used to search for the optimal cruise path on the street light spatial distribution network map with the shortest total flight distance as the optimization objective, so that the UAV can traverse all target street lights. Along the generated optimal path, image acquisition waypoints are inserted according to the calculated spatial intervals. Each waypoint includes longitude, latitude, and altitude. The discrete waypoint sequence is smoothed to generate a continuous flight trajectory, thus obtaining the UAV cruise route data.

9. A UAV target detection system based on a visual detection algorithm, characterized in that, include: At least one processing unit; for executing instructions to implement the UAV target detection method based on the vision detection algorithm as described in any one of claims 1 to 8.

Images