Target detection method and system based on real-time thermal imaging analysis of unmanned aerial vehicle

Abstract

The invention discloses a target detection method and system based on real-time thermal imaging analysis of an unmanned aerial vehicle, belongs to the technical field of image analysis, and aims to solve the problems of poor thermal imaging image processing effect, weak mode recognition capability, low detection precision and poor generalization under a complex background. According to the method, a thermal imaging sensor and an RGB camera are used for collecting a thermal imaging image and a visible light image of a target area, the thermal imaging image is preprocessed, a visible light-thermal imaging double-branch feature fusion network is constructed, a target detection model is constructed, an enhanced target feature map is reasoned, and a target detection result is obtained. According to the method, the signal-to-noise ratio of the thermal imaging image is improved, the target detail retention rate is improved, in addition, the target detection precision and the small target detection rate are improved, the mode recognition capability is superior to that of the prior art, and the method has the advantages that the method is simple and convenient to operate and high in practicability. And the method can adapt to multi-scene large-scale detection tasks.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to a target detection method and system based on real-time thermal imaging analysis of unmanned aerial vehicles (UAVs). Background Technology

[0002] The principle of drone thermal imaging analysis and detection is as follows: The drone drives the thermal imaging camera to move in real time, and the thermal imaging camera captures the infrared radiation emitted by the object itself. The captured infrared radiation is then converted into quantifiable temperature data and a visualized thermal image. Staff can analyze and detect the state of the object through the temperature data and thermal image.

[0003] Current thermal imaging analysis technology has been applied in many fields, such as equipment inspection, animal tracking, and fire monitoring. However, this technology still has many shortcomings in its use, specifically as follows: In existing UAV thermal imaging target detection technology, the image analysis stage mostly uses traditional threshold segmentation and edge detection algorithms. This analysis method performs poorly when processing thermal imaging images against complex backgrounds and is prone to losing target details. Moreover, traditional visual feature extraction methods cannot effectively capture the specific temperature distribution characteristics of thermal imaging images, leading to target positioning errors and affecting the analysis results. In addition, existing deep learning target detection models are mostly designed for visible light images. When directly transferred to thermal imaging scenarios, the lack of specialized learning of thermal signal characteristics results in weak pattern recognition capabilities, low accuracy in detecting small and dynamic targets, and poor generalization.

[0004] To address the above issues, a target detection method and system based on real-time thermal imaging analysis of unmanned aerial vehicles (UAVs) are proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a target detection method and system based on real-time thermal imaging analysis of unmanned aerial vehicles (UAVs). By using this invention, the problems of poor thermal imaging image processing performance under complex backgrounds, easy loss of target details, and the inability of traditional visual feature extraction methods to effectively capture the specific temperature distribution features of thermal imaging images, leading to target positioning errors, are solved. In addition, this invention also solves the problem that existing deep learning target detection models are mostly designed for visible light images and have weak pattern recognition capabilities, low detection accuracy, and poor generalization when directly transferred to thermal imaging scenes.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a target detection method based on real-time thermal imaging analysis of unmanned aerial vehicles, comprising the following steps:

[0007] The drone terminal is equipped with a thermal imaging sensor and an RGB camera, which are used to acquire thermal and visible light images of the target area.

[0008] Preprocess the acquired thermal imaging images;

[0009] A visible light-thermal imaging dual-branch feature fusion network was constructed to extract and fuse cross-modal features.

[0010] An object detection model is built based on the improved YOLOv11 framework to perform inference on the enhanced object feature map;

[0011] Preliminary detection results are output in real time using edge computing nodes. Then, multiple drones are coordinated to work together through a cloud server. A voting mechanism is used to fuse and verify the detection results, and finally, the target detection results are output.

[0012] Furthermore, the preprocessing process for thermal imaging images is as follows:

[0013] The acquired thermal imaging image is denoised using an algorithm that combines contrast-limited adaptive histogram equalization with Gaussian filtering. Then, the background is segmented using the Otsu thresholding algorithm, and finally, a binarized target region image and an enhanced thermal imaging image are output.

[0014] Furthermore, the process of constructing a visible light-thermal imaging dual-branch feature fusion network, extracting cross-modal features, and fusing them is as follows:

[0015] ResNet50 is used as the backbone network to perform convolution processing on visible light images and output them as visible light feature maps. The MobileNetV3-Small network is used to extract features from enhanced thermal imaging images and output them as thermal imaging feature maps. Then, the cosine similarity of the bimodal feature maps is calculated through a bidirectional alignment and fusion module to output the enhanced target feature map.

[0016] Furthermore, the process of reasoning about the enhanced target feature map is as follows:

[0017] A temperature feature-specific learning layer is added to the Backbone part of YOLOv11. The learning layer consists of two 3×3 convolutional kernels and one BN layer.

[0018] In the Neck section, a multi-scale feature pyramid is constructed, and an adaptive anchor box generation algorithm is used to generate three sets of anchor boxes. The model is then trained using a hybrid loss function.

[0019] The present invention also proposes another technical solution: a target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles (UAVs), including a UAV terminal, an edge computing node, and a cloud server;

[0020] The drone terminal includes an RGB camera that provides high-resolution scene details and color information, and a thermal imaging sensor that converts invisible infrared radiation emitted by an object into a visible thermal imaging image. The drone terminal also includes a processor and a preprocessing module that performs noise reduction and background segmentation on the acquired thermal imaging images.

[0021] The edge computing node can deploy a visible light-thermal imaging dual-branch feature fusion network and an improved YOLOv11 model to perform inference on the enhanced target feature map and output preliminary detection results;

[0022] The cloud server includes a satellite communication terminal, a GPU cluster, and a collaborative control module. The collaborative control module is used to dynamically allocate multiple drones to improve coverage and can use a voting mechanism to fuse the detection results of multiple drones for fusion verification, and then output the final target detection result.

[0023] Furthermore, during the inference process of the enhanced target feature map using the target detection model, the model is trained using a self-made thermal imaging target dataset. The sample annotation information in the target dataset includes the target bounding box, category, and temperature range. In the process of model training, random flipping, scaling, and temperature offset are used to perform data augmentation in order to improve the model performance.

[0024] Furthermore, during the model training process, the FGSM algorithm is used to generate thermal imaging-specific interference samples, which are then injected into the model training process to reduce the success rate of attacks when the model is interfered with.

[0025] Furthermore, the edge computing node is connected to the drone terminal via 5G-A millimeter wave communication. The edge computing node and the cloud server use 5G-A communication in urban areas and switch to satellite communication in remote areas to ensure stable signal transmission.

[0026] Furthermore, the final detection result includes the target category, bounding box coordinates, average temperature of the target area, and temperature confidence level.

[0027] Furthermore, the edge computing nodes are equipped with Ubuntu 20.04 LTS, TensorRT 8.6, and PyTorch 2.0, while the cloud server is equipped with CentOS 8 and a Kubernetes cluster is deployed to achieve distributed computing.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. This invention improves the algorithm in the image analysis process, thereby increasing the signal-to-noise ratio of thermal imaging images and improving the retention rate of target details, thus solving the problem of interference from complex backgrounds.

[0030] 2. This invention combines cross-modal feature fusion with a learning model, which improves the accuracy of target detection and the detection rate of small targets, and the pattern recognition capability is superior to existing technologies.

[0031] 3. The end-to-end collaborative system architecture in this invention results in a short end-to-end latency during the detection process, meeting the requirements of real-time operation of UAVs. It not only improves the coverage efficiency in multi-UAV collaborative mode, but also adapts to large-scale detection tasks in multiple scenarios. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating the method steps of the present invention;

[0033] Figure 2 This is a system flowchart of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1: Please refer to Figure 1 As shown, the target detection method based on real-time thermal imaging analysis of UAVs includes the following steps:

[0036] S1: The drone terminal is equipped with a thermal imaging sensor and an RGB camera. The thermal imaging sensor and RGB camera are used to collect thermal and visible light images of the target area, providing high-quality raw data for subsequent preprocessing and feature fusion.

[0037] The thermal imaging sensor uses a long-wave infrared thermal imaging sensor with a resolution of 640×512 pixels. While maintaining detail, it can also provide sufficient data and clearly identify targets as small as 10×10 pixels, avoiding target loss due to low resolution. The frame rate of the thermal imaging sensor is set to 15-30fps, which can be dynamically adjusted during use. For static targets, the frame rate is adjusted to 15fps to reduce the amount of data acquired, while for dynamic targets, the frame rate is adjusted to 30fps to increase the amount of data acquired, ensuring that the movement trajectory of the dynamic target can be captured. The spectral range of the thermal imaging sensor is set at 8-14μm, which can penetrate complex scenes such as fog and smoke, improving operational capabilities. The thermal imaging sensor uses a Peltier effect temperature control unit to control the monitoring range of the thermal imaging sensor within the range of -20℃ to 600℃, covering most scene requirements.

[0038] The RGB camera is a high-resolution industrial camera, set to 4K resolution to ensure more detail is captured. The RGB camera has a 24mm equivalent focal length, balancing wide-angle capabilities with enhanced detail capture. Its ISO range is 100-6400 to improve shooting capabilities in low-light and nighttime conditions, providing effective texture information for subsequent feature fusion. Furthermore, the RGB camera uses RAW format for storage, preserving more image details and avoiding texture loss caused by JPEG storage, thus facilitating smooth feature fusion.

[0039] Data acquisition strategies can be divided into single-UAV acquisition and multi-UAV acquisition, as detailed below:

[0040] Single-UAV data collection employs a combination of grid and encryption methods. In regular areas, the UAV's flight speed is controlled at 10 m / s, and the thermal imaging sensor's frame rate is set to 15 fps. The UAV follows a preset flight path, collecting one frame of dual-modal image every 5 meters at a speed of 10 m / s. In areas with suspected targets, data collection automatically switches to encryption mode, and the thermal imaging sensor's frame rate is set to 30 fps. The UAV follows a preset flight path, collecting one frame of dual-modal image every 2 meters at a speed of 5 m / s. Simultaneously, the thermal imaging sensor's depression angle is adjusted from 45° to 30° to reduce background interference and ensure high-quality data collection.

[0041] Multi-drone data collection employs a combination of grid division and dynamic adjustment. The target area is divided into multiple grid regions with an area of ​​1km×1km. Each drone is responsible for one grid region, and every two adjacent grid regions overlap by 10% to avoid blind spots and ensure the completeness of data collection. When a drone detects a target, it immediately sends a "cooperation request" to the cloud server. At this time, the cloud server immediately dispatches 1-2 adjacent drones to the target region that received the request and forms a triangulation relationship with the requesting drone. This allows the three drones to collect data on the same target from different angles. After the data collection is completed, a voting mechanism is used to fuse and verify the detection results of multiple drones, and then the final target detection result is output.

[0042] To ensure better data acquisition, a hydrophobic coating is applied to the lens of the thermal imaging sensor. This prevents poor image quality caused by rainwater adhesion during rainy weather. Furthermore, a humidity sensor is integrated inside the thermal imaging sensor. When the humidity sensor detects an ambient temperature greater than 85%, it heats the lens of the thermal imaging sensor to prevent fogging. In addition, the data cable of the thermal imaging sensor is shielded, and the drone body is grounded. This setup ensures that even if the drone encounters electromagnetic interference during the detection process, there will be no poor transmission quality or transmission errors.

[0043] Thermal imaging images acquired by thermal imaging sensors are stored in TIFF format, retaining temperature matrix data and embedding EXIF ​​information; while RGB images acquired by RGB cameras are stored in JPEG or DNG format, and then the EXIF ​​information is made consistent with the thermal imaging images to ensure matching of subsequent frames.

[0044] Each drone is equipped with a temporary cache. When communication is interrupted, the image data of the most recent 10 minutes is immediately cached. After communication is restored, the cached image data is uploaded in batches to avoid data loss and ensure data security.

[0045] S2: Preprocess the acquired thermal imaging images as follows:

[0046] An algorithm combining adaptive histogram equalization (CLAHE) and Gaussian filtering is adopted. The CLAHE parameters are set as follows: gray level range 0-255, block size 8×8 pixels, contrast limit threshold 2.0; the Gaussian filtering uses a 5×5 convolution kernel with a standard deviation σ=1.2. The image is traversed by a sliding window to suppress random noise while preserving the temperature gradient features of the target edges.

[0047] Meanwhile, an improved Otsu thresholding algorithm is proposed, which introduces the seed point selection logic of the region growing method: First, the gray-level histogram of the image is calculated to determine the initial threshold T0. The initial threshold T0 can be obtained by maximizing the inter-class variance through the Otsu algorithm. The gray value is set to be greater than T0 and the temperature value is set to be greater than the preset threshold. The pixels with the preset threshold are set as seed points. Then, the target region is expanded according to the 8-neighborhood growing rule with the seed point as the center, the target and the background are separated, and the pre-processed binarized target region image and the original enhanced image are output, where the gray-level difference between adjacent pixels is ≤5.

[0048] The specific parameters for preprocessing the acquired thermal imaging images in S2 are as follows:

[0049] At a resolution of 640×512, Adaptive Histogram Equalization (CLAHE) sets the sub-block size to 8×8 pixels, with each sub-block containing 64 pixels. This not only stabilizes the histogram statistics but also accurately improves the contrast of local targets.

[0050] The contrast limit threshold of Adaptive Histogram Equalization (CLAHE) is set to 2.0. After experimental verification with 1000 frames of thermal imaging images of different scenes, it can improve the contrast of the target area by 1.8-2.2 times, while reducing the noise intensity by 40-50%. Moreover, the gray level range of Adaptive Histogram Equalization (CLAHE) is set to 0-255, which is within the normal gray level mapping range of thermal imaging images.

[0051] To filter out more high-frequency noise, reduce edge blur rate, and reduce temperature error of edge pixels, the Gaussian filter uses a 5×5 convolution kernel with a standard deviation σ=1.2. With this setting, more than 90% of high-frequency noise can be filtered out, while the target edge blur rate is <8% and the temperature error of edge pixels is <2℃.

[0052] Calculate the gray-level histogram for the denoised thermal imaging image, iterating through all possible gray-level values ​​in the range of 0-255, and calculate the "inter-class variance" for each gray-level value as a threshold. The formula for calculating the inter-class variance is as follows:

[0053] σ²=ω0(μ0-μ)²+ω1(μ1-μ)², where ω0 and ω1 are the pixel ratios of the background and the target, μ0 and μ1 are the average gray levels of the background and the target, and μ is the average gray level of the entire image. The gray value with the largest inter-class variance is T0.

[0054] The selection criteria for seed points are: grayscale value > T0 and temperature value > preset threshold. The preset threshold can be dynamically adjusted according to different scenarios. In addition, seed point deduplication is required. The specific steps are as follows: if the distance between multiple seed points is less than 3 pixels, they are considered as the core area of ​​the same target. Only the seed point with the highest temperature is retained to avoid the phenomenon of repeated growth in subsequent areas.

[0055] The growth rules and termination conditions of the seed are as follows: Taking the seed point as the center, traverse the pixels in the 8-neighborhood (the 8-neighborhood consists of the top, bottom, left, right and four diagonal directions). If the gray level difference between adjacent pixels is ≤5, the corresponding temperature difference is ≤0.5℃, and the gray level of thermal imaging image 1 is ≈0.1℃, then the pixel is included in the target area and is used as a new seed point to continue growing. When there are no pixels in the 8-neighborhood that meet the gray level difference condition, or when the area of ​​the target area reaches the preset upper limit, the seed stops growing.

[0056] After the seed stops growing, a binarized target region image and an enhanced thermal imaging image are generated, as follows:

[0057] The target region image is binarized, with a pixel value of 255 for the target region and 0 for the background. This is used for subsequent region of interest (ROI) localization, and only pixels within the ROI are processed, reducing computational cost by 70-80%. This, in turn, reduces the computational cost of feature extraction.

[0058] Enhanced thermal imaging retains the original temperature data, removing only background noise and redundant areas for subsequent fusion with features from RGB images.

[0059] S3: Construct a visible light-thermal imaging dual-branch feature fusion network, extract and fuse cross-modal features, as detailed below:

[0060] ResNet50 is used as the backbone network. The input is a visible light image with a resolution of 640×640 pixels captured by the RGB camera on the drone. The first 5 convolutional layers extract visual features such as texture and contour, and output a visible light feature map with a resolution of 80×80 pixels and 256 channels. The convolutional kernel sizes are 7×7, 3×3, 3×3, 3×3, and 3×3, and the strides are 2, 2, 1, 1, and 1, respectively.

[0061] A lightweight convolutional network (MobileNetV3-Small) is used as input. The preprocessed thermal imaging enhancement image has a resolution of 640×640 pixels. Through depthwise separable convolution and SE attention module (compression ratio r=16), the features such as temperature distribution and thermal radiation intensity are extracted. The output is a 256-channel thermal imaging feature map with a resolution of 80×80 pixels.

[0062] Calculate the cosine similarity between corresponding pixels in the visible light feature map and the thermal imaging feature map, where the similarity threshold is ≥0.7. Use bilinear interpolation to correct the spatial position of misaligned pixels to ensure that the bimodal features match in the spatial dimension, where the similarity of misaligned pixels is <0.7.

[0063] Using a weighted summation method, with thermal imaging feature weight coefficient α=0.7 and visible light feature weight coefficient β=0.3 (optimized based on the target specificity of thermal imaging scene), the fusion formula (F_fused=α×F_thermal+β×F_visible) is used for calculation, and the final output is an enhanced target feature map with 256 channels and a resolution of 80×80 pixels.

[0064] The visible light-thermal imaging dual-branch feature fusion network constructed in S3 extracts and fuses cross-modal features using the following specific parameters:

[0065] When using ResNet50 as the backbone network, the preprocessed RGB image is 640×640 pixels with 3 channels, and needs to be normalized before input. The calculation method is: pixel value divided by 255, mean minus [0.485, 0.456, 0.406], standard deviation divided by [0.229, 0.224, 0.225]. At the same time, the normalization parameters pre-trained by ImageNet can improve the transfer effect of the pre-trained model.

[0066] When using MobileNetV3-Small as the backbone network, the preprocessed enhanced thermal imaging image first undergoes analog RGB format processing. The processing flow is as follows: converting 640×640 pixels, 1 channel, to 3 channels for input, and repeating this process 3 times consecutively. The processed enhanced thermal imaging image data also needs to be normalized before input. The normalization calculation method is as follows: divide the pixel value by 255, subtract 0.5 from the mean, divide the standard deviation by 0.5, and adapt to the grayscale distribution of the thermal imaging image.

[0067] This application uses spatial alignment to solve the positional offset problem of bimodal images. The specific reasons for the offset and the alignment method are as follows:

[0068] Reason: There is a physical distance of about 10-15cm between the RGB camera and the thermal imaging sensor on the drone. When the same target is captured, the target position in the image will be offset by 1-3 pixels. If they are not aligned, the features after fusion will be "misaligned", which will affect the detection accuracy.

[0069] Alignment method: Combining Scale Invariant Feature Transform (SIFT) feature point matching with homography matrix transformation, the specific steps are as follows:

[0070] Feature point extraction: Extract SIFT feature points from the 80×80 RGB feature map, and extract temperature gradient-based feature points from the 80×80 thermal imaging feature map;

[0071] Feature point matching: The Fast Nearest Neighbor Search (FLANN) library is used to match bimodal feature points, and corresponding points with a matching degree > 0.8 are retained to remove mismatches;

[0072] Homography matrix solution: Using the Random Sampling Consensus (RANSAC) algorithm, inliers are selected from the matching points, i.e., the error is <1 pixel. The homography matrix H (3×3) is then solved. The thermal imaging feature map is transformed by H and aligned to the coordinate system of the RGB feature map, with an alignment error ≤0.5 pixels.

[0073] Effective fused pixels are selected using cosine similarity calculation, and the calculation formula is as follows:

[0074] ;

[0075] in This is the aligned RGB feature map pixel vector with a dimension of 256; It is a pixel vector of thermal imaging feature map with a dimension of 256.

[0076] The similarity threshold is set to greater than or equal to 0.7. When the similarity of pixels is ≥0.7, it means that the pixel is a "valid feature" in both modes. When the similarity is <0.7, the pixel is a "valid feature in one mode" and needs to be corrected by interpolation to avoid fusion interference.

[0077] The specific interpolation steps are as follows: Bilinear interpolation is used to calculate the interpolated feature value based on the four pixel values ​​around the pixel with a similarity of ≥0.7, to ensure the continuity of the feature map.

[0078] The dual-modal weights are dynamically adjusted using a weighted fusion method, as shown in the following formula: ,in ;

[0079] Weight determination: α and β are adaptively adjusted based on the scenario.

[0080] Low light / foggy scenes: α=0.8, β=0.2, characterized by: loss of RGB texture, dependent on thermal imaging temperature features;

[0081] Normal lighting scene: α=0.7, β=0.3, characterized by: thermal imaging temperature features as the main feature, RGB textures supplementing details;

[0082] Validation data: In a fusion experiment with 1000 frames of different scenes, when the fusion feature map with adaptive weights was used for detection, the mean accuracy (mAP) was improved by 5-8% compared with the fixed weights (α=0.7, β=0.3).

[0083] Output: Enhanced target feature map with 256 channels and 80×80 resolution. This feature map contains both thermal imaging branch information in temperature distribution and RGB branch information in texture contour, providing high-quality features for subsequent detection and inference.

[0084] S4: Based on the improved YOLOv11 framework, a target detection model is built to perform inference on the enhanced target feature map, as detailed below:

[0085] A temperature feature-specific learning layer is added to the Backbone part of YOLOv11. This layer consists of two 3×3 convolutional kernels and one BN layer. The convolutional kernel stride is 1, and the parameters of the convolutional kernel are optimized through training. It specifically learns the temperature gradient changes in the thermal imaging feature map, which enhances the ability to recognize patterns of thermally specific targets.

[0086] In the Neck section, a multi-scale feature pyramid is constructed with three feature levels, corresponding to small, medium, and large-sized targets respectively. At the same time, an adaptive anchor box generation algorithm is adopted based on the target size statistics in the training set to improve the matching accuracy of targets of different scales.

[0087] In the process of using the object detection model to infer the enhanced object feature map, a self-made thermal imaging object dataset is used to train the model. The sample annotation information in the object dataset includes the object bounding box, category and temperature range. In addition, random flipping, scaling and temperature offset are used to augment the data during the model training process in order to improve the model performance.

[0088] Then, the edge computing nodes are used to output the preliminary detection results in real time. Then, multiple drones are scheduled to work together through the cloud server, and a voting mechanism is used to fuse and verify the detection results, and finally the target detection results are output.

[0089] In addition, during the model training process, the FGSM algorithm is used to generate thermal imaging-specific interference samples, which are then injected into the model training process to reduce the success rate of attacks when the model is interfered with.

[0090] In S4, an object detection model is built based on the improved YOLOv11 framework. The specific parameters for inference on the enhanced object feature map are as follows:

[0091] After the Conv4 layer of YOLOv11Backbone, i.e. after the output of the 40×40 feature map, a temperature feature-specific learning layer is added. The temperature feature-specific learning layer consists of two 3×3 convolutional kernels and one BN layer + one SiLU activation function.

[0092] Convolution kernel parameters: 3×3, number = current number of channels, i.e. Conv4 output has 512 channels, so the number of convolution kernels in this layer is 512; stride = 1; padding = 1, to ensure that the output resolution is consistent with the input, both are 40×40.

[0093] The purpose of the Batch Normalization (BN) layer is to perform batch normalization on the data, avoid feature distribution shift after adding a new layer, and accelerate training convergence.

[0094] SiLU activation function: Compared to ReLU, SiLU has a gradient in the negative region, which can better learn subtle changes in temperature characteristics.

[0095] The temperature feature-specific learning layer is specifically designed to learn temperature gradient patterns in thermal imaging features. It captures the temperature distribution patterns of the target area through convolutional kernels, encodes temperature features into high-dimensional feature vectors, and fuses them with the original visual features to improve the recognition ability of temperature-related targets.

[0096] The Neck section constructs a multi-scale feature pyramid and generates an adaptive anchor box algorithm. The specific steps are as follows: The original YOLOv11 Neck outputs three feature maps with scales of 80×80, 40×40, and 20×20. This application adds an ultra-small scale feature layer of 160×160 for small targets in thermal imaging.

[0097] From the Conv2 layer of the Backbone, which outputs a 160×160, 256-channel feature map, a branch is drawn out, and the channels are compressed to 128 through a 1×1 convolution; then it is concatenated with the feature map of the Neck's 80×80 feature map after being sampled by 2 times, to form an ultra-small scale feature layer of 160×160, 256 channels.

[0098] Optimized results: The ultra-small scale feature layer can capture small targets smaller than 10×10 pixels, and the detection efficiency is improved.

[0099] Adaptive anchor box generation steps:

[0100] The aspect ratio (w / h) and dimensions (w×h) of all targets in the self-made thermal imaging dataset are statistically analyzed to form a target size library;

[0101] The K-means clustering algorithm (K=3, corresponding to small, medium, and large targets) is used to cluster the target size library to generate anchor boxes adapted to thermal imaging scenarios.

[0102] Optimized results: The intersection-union ratio (IOU) between the adaptive anchor box and the target increased from 65% to 85% of the original predefined anchor box, and the bounding box positioning error was reduced by 30%.

[0103] The formula for calculating the hybrid loss is as follows: ;

[0104] in: For bounding box localization loss, considering the overlap of bounding boxes, the distance between center points, and the aspect ratio, the localization accuracy is improved by 20% compared to the IoU loss.

[0105] The cross-entropy loss is used for target class prediction. FocalLoss is used to address class imbalance, such as the problem that there are far more background pixels than target pixels in thermal imaging images. FocalLoss has γ=2 and α=0.25.

[0106] For temperature loss, the MAE (mean absolute error) between the target temperature range predicted by the model and the actual temperature range is calculated. Where N is the number of pixels in the target region. To predict temperature, This is the actual temperature;

[0107] The temperature loss weight is set to 0.3, which minimizes the temperature prediction error without affecting the optimization of localization and classification losses.

[0108] After calculation using the mixed loss function, the temperature prediction error decreased from ±5℃ to ±2℃, and the misclassification rate of temperature-related categories decreased from 25% to 8%.

[0109] We created 150,000 dual-modal images based on different scenarios, covering multiple different scenarios, and used the LabelMe tool to annotate each image. The annotation parameters included target bounding box, target category, temperature range, and environmental parameters. At the same time, we used random flipping and temperature enhancement to perform spatial enhancement, thereby realizing a data augmentation strategy to improve generalization.

[0110] In addition, this application also improves anti-jamming capability through FGSM adversarial training and uses the FGSM algorithm to generate thermal imaging-specific interference samples, as detailed below: ;

[0111] in Original image; The disturbance intensity; The gradient of the loss function; This is a real label.

[0112] The adversarial training process is as follows: Every 10 rounds of training, 10,000 frames of FGSM interference samples are generated; the interference samples are mixed with normal samples at a ratio of 1:10 for subsequent training.

[0113] Post-training results: The model's attack success rate under interference decreased from 35% to 8%.

[0114] Then, TensorRT 8.6 is used to perform FP16 quantization on the trained model, and the inference is accelerated by using the TensorRT engine deployed on the edge computing node in a "dynamic batch processing" manner. After the inference is completed, the preliminary detection results are output.

[0115] S5: Utilizes edge computing nodes to output preliminary detection results in real time, then schedules multiple drones to work collaboratively through a cloud server, and uses a voting mechanism to fuse and verify the detection results, thereby outputting the final target detection results.

[0116] Example 2, as Figure 2 As shown, the target detection system based on real-time thermal imaging analysis of UAVs includes a UAV terminal, an edge computing node, and a cloud server.

[0117] The drone terminal includes an RGB camera that provides high-resolution scene details and color information, and a thermal imaging sensor that converts invisible infrared radiation emitted by objects into visible thermal images. The drone terminal also includes a processor and a preprocessing module that performs noise reduction and background segmentation on the acquired thermal images.

[0118] Edge computing nodes can deploy visible light-thermal imaging dual-branch feature fusion networks and improved YOLOv11 models to perform inference on enhanced target feature maps and output preliminary detection results. The edge computing nodes are equipped with Ubuntu 20.04LTS, TensorRT 8.6 and PyTorch 2.0, while the cloud servers are equipped with CentOS 8 and deployed with Kubernetes clusters to achieve distributed computing.

[0119] The cloud server includes a satellite communication terminal, a GPU cluster, and a collaborative control module. The collaborative control module is used to dynamically allocate multiple drones to improve coverage and can use a voting mechanism to fuse the detection results of multiple drones for fusion verification, thereby outputting the final target detection result. The final detection result includes the target category, bounding box coordinates, average temperature of the target area, and temperature confidence level.

[0120] The cloud server also includes a blockchain evidence storage module, which generates a unique hash value for each test result and stores it in a distributed ledger, and supports authorized access from multiple nodes.

[0121] Edge computing nodes and drone terminals are connected via 5G-A millimeter-wave communication. Edge computing nodes and cloud servers use 5G-A communication in urban areas and switch to satellite communication in remote areas to ensure stable signal transmission.

[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A target detection method based on real-time thermal imaging analysis of unmanned aerial vehicles, characterized in that, Includes the following steps: The drone terminal is equipped with a thermal imaging sensor and an RGB camera, which are used to acquire thermal and visible light images of the target area. Preprocess the acquired thermal imaging images; A visible light-thermal imaging dual-branch feature fusion network was constructed to extract and fuse cross-modal features. An object detection model is built based on the improved YOLOv11 framework to perform inference on the enhanced object feature map; Preliminary detection results are output in real time using edge computing nodes. Then, multiple drones are coordinated to work together through a cloud server. A voting mechanism is used to fuse and verify the detection results, and finally, the target detection results are output.

2. The target detection method based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 1, characterized in that, The process of preprocessing thermal imaging images is as follows: The acquired thermal imaging image is denoised using an algorithm that combines contrast-limited adaptive histogram equalization with Gaussian filtering. Then, the background is segmented using the Otsu thresholding algorithm, and finally, a binarized target region image and an enhanced thermal imaging image are output.

3. The target detection method based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 2, characterized in that, The process of constructing a visible light-thermal imaging dual-branch feature fusion network, extracting cross-modal features, and fusing them is as follows: ResNet50 is used as the backbone network to perform convolution processing on visible light images and output them as visible light feature maps. The MobileNetV3-Small network is used to extract features from enhanced thermal imaging images and output them as thermal imaging feature maps. Then, the cosine similarity of the bimodal feature maps is calculated through a bidirectional alignment and fusion module to output the enhanced target feature map.

4. The target detection method based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 3, characterized in that, The process of reasoning about the enhanced target feature map is as follows: A temperature feature-specific learning layer is added to the Backbone part of YOLOv11. The learning layer consists of two 3×3 convolutional kernels and one BN layer. In the Neck section, a multi-scale feature pyramid is constructed, and an adaptive anchor box generation algorithm is used to generate three sets of anchor boxes. The model is then trained using a hybrid loss function.

5. A target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles, applied to the method described in any one of claims 1-4, characterized in that, This includes drone terminals, edge computing nodes, and cloud servers; The drone terminal includes an RGB camera that provides high-resolution scene details and color information, and a thermal imaging sensor that converts invisible infrared radiation emitted by an object into a visible thermal imaging image. The drone terminal also includes a processor and a preprocessing module that performs noise reduction and background segmentation on the acquired thermal imaging images. The edge computing node can deploy a visible light-thermal imaging dual-branch feature fusion network and an improved YOLOv11 model to perform inference on the enhanced target feature map and output preliminary detection results; The cloud server includes a satellite communication terminal, a GPU cluster, and a collaborative control module. The collaborative control module is used to dynamically allocate multiple drones to improve coverage and can use a voting mechanism to fuse the detection results of multiple drones for fusion verification, and then output the final target detection result.

6. The target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 5, characterized in that, In the process of using the object detection model to infer the enhanced object feature map, a self-made thermal imaging object dataset is used to train the model. The sample annotation information in the object dataset includes the object bounding box, category and temperature range. In addition, random flipping, scaling and temperature offset are used to augment the data during the model training process in order to improve the model performance.

7. The target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 6, characterized in that, During the model training process, the FGSM algorithm is used to generate thermal imaging-specific interference samples. These samples are then injected into the model training process to reduce the success rate of attacks when the model is interfered with.

8. The target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 7, characterized in that, The edge computing node is connected to the drone terminal via 5G-A millimeter wave communication. The edge computing node and the cloud server use 5G-A communication in urban areas and switch to satellite communication in remote areas to ensure stable signal transmission.

9. The target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 8, characterized in that, The final detection result includes the target category, bounding box coordinates, average temperature of the target area, and temperature confidence level. The cloud server also includes a blockchain evidence storage module, which generates a unique hash value for each detection result and stores it in a distributed ledger, and supports authorized access from multiple nodes.

10. The target detection system based on real-time thermal imaging analysis of unmanned aerial vehicles according to claim 9, characterized in that, The edge computing nodes are equipped with Ubuntu 20.04 LTS, TensorRT 8.6 and PyTorch 2.0, while the cloud server is equipped with CentOS 8 and a Kubernetes cluster is deployed to achieve distributed computing.

Images