Weak light aerial photography small target detection method and system based on low light enhancement multi-mode fusion

Abstract

The invention discloses a low light enhancement multimode fusion weak light aerial photography small target detection method and system, and relates to the technical field of computer vision, and the method comprises the following steps: S1, inputting an original visible light image into a pre-trained low light enhancement network, and obtaining an enhanced visible light image; s2, inputting the enhanced visible light image and the original infrared image into a pre-trained cross-modal homography registration network to obtain an aligned infrared image; wherein the original infrared image and the original visible light image are synchronously obtained; s3, inputting the aligned infrared image and the enhanced visible light image into a pre-trained symmetric compact pixel-level fusion network to obtain a final fusion feature map; and S4, inputting the final fusion feature map into a pre-trained improved YOLOv5-Lite detection network to obtain a small target detection result. According to the invention, the detection precision, robustness and real-time performance of weak light aerial photography small target detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a low-light enhanced multi-mode fusion method and system for detecting small targets in low-light aerial photography. Background Technology

[0002] Currently, drones rely heavily on visible light and infrared sensors for ground target detection in tasks such as nighttime patrols, emergency search and rescue, and border surveillance. Existing methods typically use a single visible light camera or infrared camera for sensing, or directly perform convolutional feature cascade fusion on visible light and infrared images in multimodal detection.

[0003] However, the signal-to-noise ratio of a single visible light camera decreases significantly under low-light conditions, resulting in the loss of a large amount of image details and making it difficult to distinguish small targets. Although a single infrared camera is not affected by changes in illumination, it lacks texture and color information, resulting in limited positioning accuracy. Furthermore, existing multimodal methods generally assume that the visible light and infrared images are strictly aligned, ignoring the intermodal parallax problem caused by changes in viewpoint and attitude during UAV flight, and failing to effectively preprocess low-light degradation, leading to insufficient reliability of feature fusion.

[0004] Therefore, how to provide a method and system for detecting small targets in low-light aerial photography that can improve the detection accuracy, robustness and real-time performance of small target detection in low-light aerial photography is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a low-light enhanced multi-mode fusion method and system for detecting small targets in aerial photography.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a low-light aerial photography small target detection method with low-light enhancement multi-mode fusion is provided, including the following steps: S1: Input the original visible light image into a pre-trained low-light enhancement network to obtain an enhanced visible light image; S2: Input the enhanced visible light image and the original infrared image into a pre-trained cross-modal homography registration network to obtain an aligned infrared image; wherein, the original infrared image and the original visible light image are obtained simultaneously; S3: Input the aligned infrared image and the enhanced visible light image into a pre-trained symmetric compact pixel-level fusion network to obtain the final fused feature map; S4: Input the final fused feature map into the pre-trained improved YOLOv5-Lite detection network to obtain small target detection results.

[0007] Preferably, the low-light enhancement network employs a lightweight convolutional network based on Retinex theory.

[0008] Preferably, S2 specifically includes the following steps: The enhanced visible light image and the original infrared image are input into a pre-trained lightweight homography estimator to obtain the homography matrix; The original infrared image is subjected to perspective transformation using the homography matrix to obtain the aligned infrared image.

[0009] Preferably, the expression for the perspective transformation is: ; In the formula, This refers to the aligned infrared image; This indicates that the aligned infrared image is in coordinates Pixel value at; This refers to the original infrared image; This indicates that the original infrared image is located in coordinates. Pixel value at; Represents the homography matrix; The coordinates of a pixel in the original infrared image are represented by ; T represents the transpose.

[0010] Preferably, S3 specifically includes the following steps: After normalizing the aligned infrared image and the enhanced visible light image, they are symmetrically stitched together in the channel dimension to obtain an initial fused feature map. The initial fused feature map is input into a spatial-channel attention filter kernel to obtain an intermediate fused feature map; The intermediate fused feature map is subjected to 1*1 convolution dimensionality reduction to obtain the final fused feature map.

[0011] Preferably, the improved YOLOv5-Lite detection network removes the original Focus module, inserts a spatial-channel filter layer between the Backbone and Neck, and applies cascaded supervision to the P2–P5 multi-scale output.

[0012] Preferably, the low-light enhancement network, the cross-modal homography registration network, the symmetric compact pixel-level fusion network, and the improved YOLOv5-Lite detection network are trained using the total loss function; The expression for the total loss function is: ; In the formula, Represents the total loss function; Indicates detection loss; Indicates enhancement-reconstruction loss; This indicates a loss of uniformity in illumination; Indicates registration loss; , , This represents the weighting coefficient.

[0013] Preferably, the expression for the detection loss is: ; In the formula, Indicates classification loss; This represents the bounding box regression loss; Indicates the target confidence loss; The expression for the enhancement-reconstruction loss is: ; In the formula, and Indicates the weighting coefficient; This represents the reflectance map obtained by illumination map decomposition of the original visible light training image through a low-light enhancement network. This represents the true reflectance of the original visible light training image; Indicates the use of super-resolution auxiliary branches The high-resolution visible light training image obtained by upsampling and reconstructing the original visible light training image; This represents the high-resolution supervision image corresponding to the original visible light training image.

[0014] Preferably, the expression for the registration loss is: ; In the formula, This represents the enhanced visible light training image obtained by processing the original visible light training image using a low-light enhancement network. This represents the aligned infrared training image obtained by processing the original infrared training image through a cross-modal homography registration network. Indicates the balance coefficient; This indicates the calculation of the gradient of the homography matrix H.

[0015] In a second aspect, a low-light aerial photography small target detection system with low-light enhancement multi-mode fusion is provided to implement the low-light aerial photography small target detection method described in the first aspect, including an image acquisition unit, a computing unit and an image transmission unit; The image acquisition unit is used to simultaneously acquire the original visible light image and the original infrared image; The computing unit is used to execute the low-light aerial small target detection method according to any one of claims 1-9; The image transmission unit is used to transmit the small target detection results output by the computing unit back to the ground station in real time.

[0016] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a low-light enhanced multi-mode fusion method and system for detecting small targets in low-light aerial photography, which can achieve the following beneficial technical effects: 1) This invention trains the low-light enhancement network and the super-resolution auxiliary branch in a coordinated manner, which significantly restores the details and contrast of visible light images and effectively suppresses low-light noise.

[0017] 2) The lightweight homography estimator introduced in this invention can significantly reduce the geometric errors caused by UAV attitude changes and parallax, and make visible light and infrared images highly consistent at the pixel level.

[0018] 3) The symmetrical compact pixel-level fusion network of the present invention can fully exploit the complementary information of the two modes, which significantly improves the accuracy and stability of detection.

[0019] 4) The improved YOLOv5-Lite detection network of this invention removes the traditional Focus module and adds a spatial-channel filtering layer, which not only retains high-resolution input but also effectively suppresses background interference, significantly improving the ability to identify small targets.

[0020] 5) Thanks to super-resolution assisted branch pruning and overall lightweight design, this invention maintains full functionality while still having good real-time inference speed, meeting the requirements of UAV edge computing.

[0021] In summary, this invention balances detection accuracy, robustness, and real-time performance in low-light aerial photography scenarios, demonstrating outstanding comprehensive technical advantages. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 The flowchart illustrates a low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography, as provided by this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Firstly, such as Figure 1 As shown in the figure, this invention discloses a low-light enhanced multi-mode fusion method for detecting small targets in aerial photography, including the following steps: S1: Input the original visible light image into a pre-trained low-light enhancement network to obtain an enhanced visible light image; In one or more embodiments, the low-light enhancement network employs a lightweight convolutional network based on Retinex theory.

[0026] It is understandable that this invention introduces a super-resolution auxiliary branch during the training phase. Super-resolution auxiliary branch This is used to upsample and reconstruct enhanced visible light training images to obtain high-resolution visible light training images, and the super-resolution auxiliary branch is pruned during the deployment and inference phase. This ensures the real-time performance of edge computing.

[0027] A lightweight convolutional network based on Retinex theory is used to perform illumination map decomposition and noise suppression on the original visible light training images; illumination map decomposition transforms the original visible light training images... It is decomposed into an illumination map L and a reflection map R; where, .

[0028] S2: Input the enhanced visible light image and the original infrared image into a pre-trained cross-modal homography registration network to obtain an aligned infrared image; wherein, the original infrared image and the original visible light image are obtained simultaneously; In one or more embodiments, S2 specifically includes the following steps: The enhanced visible light image and the original infrared image are input into a pre-trained lightweight homography estimator to obtain the homography matrix; The original infrared image is subjected to perspective transformation using the homography matrix to obtain the aligned infrared image.

[0029] In one or more embodiments, the expression for the perspective transformation is: ; In the formula, This refers to the aligned infrared image; This indicates that the aligned infrared image is in coordinates Pixel value at; This refers to the original infrared image; This indicates that the original infrared image is located in coordinates. Pixel value at; Represents the homography matrix; The coordinates of a pixel in the original infrared image are represented by ; T represents the transpose.

[0030] It should be noted that the size of the homography matrix is ​​3*3.

[0031] It is understandable that the design of a lightweight homography estimator and perspective transformation can eliminate displacement differences caused by flight jitter and 3D parallax.

[0032] S3: Input the aligned infrared image and the enhanced visible light image into a pre-trained symmetric compact pixel-level fusion network to obtain the final fused feature map; In one or more embodiments, S3 specifically includes the following steps: After normalizing the aligned infrared image and the enhanced visible light image, they are symmetrically stitched together in the channel dimension to obtain an initial fused feature map. It is understood that the normalization is to normalize the pixel values ​​of each pixel in the aligned infrared image and the enhanced visible light image to [0,1].

[0033] The initial fused feature map is input into a spatial-channel attention filter kernel to obtain an intermediate fused feature map; It is understood that the spatial-channel attention filter kernel is used to extract complementary features and suppress redundant noise.

[0034] The intermediate fused feature map is subjected to 1*1 convolution dimensionality reduction to obtain the final fused feature map.

[0035] Understandably, the final fused feature map has the same size as the original infrared image and the original visible light image, ensuring the stability of the computational load of the subsequent detection network.

[0036] S4: Input the final fused feature map into the pre-trained improved YOLOv5-Lite detection network to obtain small target detection results.

[0037] It is understandable that small target detection results include target category, location, and confidence information.

[0038] In one or more embodiments, the improved YOLOv5-Lite detection network removes the original Focus module and directly receives high-resolution features; a spatial-channel filtering layer is inserted between the Backbone and Neck to suppress background interference and amplify the response of small targets; and cascaded supervision is applied to the P2–P5 multi-scale output.

[0039] In one or more embodiments, the low-light enhancement network, the cross-modal homography registration network, the symmetric compact pixel-level fusion network, and the improved YOLOv5-Lite detection network are trained using a total loss function; The expression for the total loss function is: ; In the formula, Represents the total loss function; Indicates detection loss; Indicates enhancement-reconstruction loss; This indicates a loss of uniformity in illumination; Indicates registration loss; , , This represents the weighting coefficient.

[0040] The expression for the illumination uniformity loss is:

[0041] In the formula, This represents the original visible light training image (low light input). This represents the reflectance image obtained by decomposition using a low-light enhancement network. This represents the illumination map predicted by the low-light enhancement network; This indicates pixel-by-pixel multiplication; They represent in / Gradient operator for direction; For pixel position, Total number of pixels; These are the weighting coefficients; In one or more embodiments, the expression for the detection loss is: ; In the formula, Indicates classification loss; This represents the bounding box regression loss; Indicates the target confidence loss; The expression for the enhancement-reconstruction loss is: ; In the formula, and Indicates the weighting coefficient; This represents the reflectance map obtained by illumination map decomposition of the original visible light training image through a low-light enhancement network. This represents the true reflectance of the original visible light training image; Indicates the use of super-resolution auxiliary branches The high-resolution visible light training image obtained by upsampling and reconstructing the original visible light training image; This represents the high-resolution supervision image corresponding to the original visible light training image.

[0042] In one or more embodiments, the registration loss is expressed as: ; In the formula, This represents the enhanced visible light training image obtained by processing the original visible light training image using a low-light enhancement network. This represents the aligned infrared training image obtained by processing the original infrared training image through a cross-modal homography registration network. Indicates the balance coefficient; This indicates the calculation of the gradient of the homography matrix H.

[0043] During the deployment phase, this invention prunes the super-resolution auxiliary branch. It retains only the enhancement-registration-fusion-detection main link, and is equipped with NVIDIA Jetson Orin NX at an input resolution of 640 × 640, which can meet the strict real-time requirements of drone edge devices.

[0044] All networks in this invention employ a lightweight design: the low-light enhancement network uses a multi-layer pyramid structure to accommodate details at different scales; the lightweight homography estimator achieves low-latency registration through depthwise separable convolutions; the symmetric compact pixel-level fusion network applies attention mechanisms simultaneously in the spatial and channel dimensions to highlight effective features; and the improved YOLOv5-Lite detection network removes redundant downsampling structures and inserts spatial-channel filtering layers to enhance sensitivity to small targets. The hyperparameters of each network, such as the number of channels, learning rate, and loss weights, can be adjusted according to different hardware resources and application scenarios to achieve the optimal balance between detection accuracy and inference speed.

[0045] In a second aspect, embodiments of the present invention disclose a low-light enhanced multi-mode fusion small target detection system for aerial photography, used to implement the small target detection method for aerial photography described in the first aspect, including an image acquisition unit, a calculation unit and an image transmission unit; The image acquisition unit is used to simultaneously acquire the original visible light image and the original infrared image; In one or more embodiments, the image acquisition unit includes a visible light camera and an infrared camera coaxially mounted on a three-axis stabilized gimbal; The visible light camera is used to acquire visible light images; The infrared camera is used to acquire infrared images.

[0046] In one or more embodiments, the visible light camera and the infrared camera have collinear optical centers and overlapping fields of view to improve the geometric consistency of multimodal data.

[0047] The computing unit is used to execute the low-light aerial small target detection method according to any one of claims 1-9; In one or more embodiments, the computing unit is an onboard SoC computing unit; the onboard SoC computing unit is a processor or accelerator for running pre-trained low-light enhancement networks, pre-trained cross-modal homography registration networks, pre-trained symmetric compact pixel-level fusion networks, and pre-trained improved YOLOv5-Lite detection networks. The image transmission unit is used to transmit the small target detection results output by the computing unit back to the ground station in real time.

[0048] In one or more embodiments, the low-light aerial small target detection system further includes a power management unit, which is used to supply power to the image acquisition unit, the computing unit and the image transmission unit and manage power consumption.

[0049] Finally, the invention was fully validated on a multi-source low-light aerial photography dataset. The validation results show that, compared with traditional detection frameworks that do not employ the method of this invention, the invention demonstrates significant advantages in key indicators such as target detection rate, positioning accuracy, and false alarm control. Simultaneously, it maintains a stable real-time inference speed on edge SoC platforms, meeting the comprehensive requirements of accuracy, robustness, and timeliness for practical tasks such as nighttime patrols, emergency search and rescue, and border surveillance.

[0050] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0051] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A low-light enhanced multi-mode fusion method for detecting small targets in aerial photography, characterized in that, Includes the following steps: S1: Input the original visible light image into a pre-trained low-light enhancement network to obtain an enhanced visible light image; S2: Input the enhanced visible light image and the original infrared image into a pre-trained cross-modal homography registration network to obtain an aligned infrared image; wherein, the original infrared image and the original visible light image are obtained simultaneously; S3: Input the aligned infrared image and the enhanced visible light image into a pre-trained symmetric compact pixel-level fusion network to obtain the final fused feature map; S4: Input the final fused feature map into the pre-trained improved YOLOv5-Lite detection network to obtain small target detection results.

2. The low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography according to claim 1, characterized in that, The low-light enhancement network employs a lightweight convolutional network based on Retinex theory.

3. The low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography according to claim 1, characterized in that, S2 specifically includes the following steps: The enhanced visible light image and the original infrared image are input into a pre-trained lightweight homography estimator to obtain the homography matrix; The original infrared image is subjected to perspective transformation using the homography matrix to obtain the aligned infrared image.

4. The low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography according to claim 3, characterized in that, The expression for the perspective transformation is: ； In the formula, This refers to the aligned infrared image; This indicates that the aligned infrared image is in coordinates Pixel value at; This refers to the original infrared image; This indicates that the original infrared image is located in coordinates. Pixel value at; Represents the homography matrix; The coordinates of a pixel in the original infrared image are represented by ; T represents the transpose.

5. The low-light aerial photography small target detection method based on low-light enhanced multi-mode fusion according to claim 4, characterized in that, S3 specifically includes the following steps: After normalizing the aligned infrared image and the enhanced visible light image, they are symmetrically stitched together in the channel dimension to obtain an initial fused feature map. The initial fused feature map is input into a spatial-channel attention filter kernel to obtain an intermediate fused feature map; The intermediate fused feature map is subjected to 1*1 convolution dimensionality reduction to obtain the final fused feature map.

6. The low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography according to claim 5, characterized in that, The improved YOLOv5-Lite detection network removes the original Focus module, inserts a spatial-channel filter layer between the Backbone and Neck, and applies cascaded supervision to the P2–P5 multi-scale output.

7. The low-light enhanced multi-mode fusion method for detecting small targets in low-light aerial photography according to claim 6, characterized in that, The low-light enhancement network, the cross-modal homography registration network, the symmetric compact pixel-level fusion network, and the improved YOLOv5-Lite detection network are trained using the total loss function. The expression for the total loss function is: ； In the formula, Represents the total loss function; Indicates detection loss; Indicates enhancement-reconstruction loss; This indicates a loss of uniformity in illumination; Indicates registration loss; , , This represents the weighting coefficient.

8. The low-light aerial photography small target detection method based on low-light enhanced multi-mode fusion according to claim 7, characterized in that: The expression for the detection loss is: ； In the formula, Indicates classification loss; This represents the bounding box regression loss; Indicates the target confidence loss; The expression for the enhancement-reconstruction loss is: ； In the formula, and Indicates the weighting coefficient; This represents the reflectance map obtained by illumination map decomposition of the original visible light training image through a low-light enhancement network. This represents the true reflectance of the original visible light training image; Indicates the use of super-resolution auxiliary branches The high-resolution visible light training image obtained by upsampling and reconstructing the original visible light training image; This represents the high-resolution supervision image corresponding to the original visible light training image.

9. The low-light aerial photography small target detection method based on low-light enhanced multi-mode fusion according to claim 7, characterized in that: The expression for the registration loss is: ； In the formula, This represents the enhanced visible light training image obtained by processing the original visible light training image using a low-light enhancement network. This represents the aligned infrared training image obtained by processing the original infrared training image through a cross-modal homography registration network. Indicates the balance coefficient; This indicates the calculation of the gradient of the homography matrix H.

10. A low-light enhanced multi-mode fusion system for detecting small targets in aerial photography, characterized in that: The method for detecting small targets in low-light aerial photography according to any one of claims 1-9 includes an image acquisition unit, a calculation unit, and an image transmission unit. The image acquisition unit is used to simultaneously acquire the original visible light image and the original infrared image; The computing unit is used to execute the low-light aerial small target detection method according to any one of claims 1-9; The image transmission unit is used to transmit the small target detection results output by the computing unit back to the ground station in real time.

Images