Multi-modal fusion unmanned aerial vehicle remote sensing image target detection method and system

Abstract

The invention discloses a multi-modal fusion unmanned aerial vehicle remote sensing image target detection method and system, aims to improve the target detection precision under a low illumination condition and reduce the calculation overhead, and is particularly suitable for unmanned aerial vehicle remote sensing image processing. The method comprises the following main steps: S1, carrying out denoising, standardization and size adjustment on an input remote sensing image, ensuring image quality and consistency, and stabilizing subsequent processing steps; and S2, based on a Retinex principle, enhancing the low-illumination image through a double-branch network, and improving details and contrast of the image. And S3, carrying out feature interaction fusion on the enhanced RGB image and the infrared image by adopting a Transform model based on a self-attention mechanism, capturing complementary information of different modes, and generating fusion features for target detection. And S4, target regression and classification are carried out by adopting a multi-scale detection head based on FPN and PAN structures, and the detection precision of targets of different sizes is enhanced. Through multi-modal feature fusion, adaptive anchor frame generation and multi-scale detection, the target detection precision in low-illumination and complex environments is effectively improved, the calculation overhead is low, and the method is suitable for target detection tasks of real-time unmanned aerial vehicle remote sensing images. Experimental results show that the method is excellent in performance on VisDrone and LLVIP data sets, and the target detection precision is remarkably improved especially under the low illumination condition.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image target detection, and particularly to a multi-modal fusion remote sensing image target detection method and system thereof. The method is applicable to target detection tasks under low light conditions, can effectively improve the target detection accuracy by fusing the features of RGB images and infrared images, and has a low computational cost, and is applicable to real-time target detection of unmanned aerial vehicle (UAV) remote sensing images. Background Art

[0002] In recent years, with the rapid development of UAV technology, target detection technology has been widely applied in the field of UAV remote sensing. Especially in multiple industries such as power inspection, emergency rescue, smart agriculture and forestry, it plays an important role. As a flexible and efficient remote sensing platform, UAVs can carry various sensors, such as infrared cameras, depth cameras, lidar, etc., providing multi-modal image information. These different modal image information can perceive the same scene from different angles, thus effectively improving the accuracy and robustness of target detection.

[0003] In multi-modal remote sensing image target detection, fusing data from multiple sensors can obtain richer scene information. In particular, the combination of infrared images and visible light images can effectively make up for the limitations of a single modality under different conditions. Infrared images still have strong target detection capabilities under low light and bad weather conditions, while visible light images show better detail information under high light conditions. Therefore, reasonable multi-modal image fusion can improve the accuracy of target detection, especially in complex environments and variable lighting conditions, further enhancing the adaptability and reliability of UAV remote sensing image processing.

[0004] However, most of the existing fusion algorithms for visible light images and infrared images have significant limitations in low light environments. Specifically, under low light conditions, traditional image processing methods often fail to fully exploit the potential of the visible light modality, resulting in the neglect of its feature information during the fusion process, which in turn affects the final target detection accuracy. Although some deep learning-based methods have made partial progress, they still face challenges such as how to effectively fuse multi-modal image features under low light, improve target detection performance, and reduce computational cost.

[0005] Therefore, in order to improve the generality and robustness of UAV remote sensing image target detection, especially in low light and complex environments, it is particularly important to study multi-modal image fusion target detection technology for low light environments. This can not only enhance the adaptability of the detection algorithm in various environments, but also provide more efficient and accurate support for UAVs when performing various tasks (such as night inspections, post-disaster assessments, etc.). Summary of the Invention

[0006] The present invention provides a multi-modal fusion remote sensing image target detection method and system for unmanned aerial vehicles, aiming to solve the problems of low target detection accuracy and susceptibility to interference under low light conditions. Traditional remote sensing image processing methods are difficult to effectively improve the detection accuracy under low light and complex environments, and at the same time consume a large amount of computing resources. By combining low-light image enhancement, multi-modal image fusion and multi-scale target detection technologies, the present invention proposes an efficient and low-computation-overhead target detection method, which is suitable for real-time target detection tasks of unmanned aerial vehicle remote sensing images.

[0007] In a first aspect, an embodiment of the present invention provides a multi-modal fusion remote sensing image target detection method for unmanned aerial vehicles, including:

[0008] S1: Denoise, standardize and adjust the size of the input remote sensing image to ensure image quality and consistency, and stabilize subsequent processing steps.

[0009] S2: Based on the Retinex principle, enhance the low-light image through a dual-branch network to improve the details and contrast of the image.

[0010] S3: Adopt a Transformer model based on the self-attention mechanism to perform feature interaction and fusion on the enhanced RGB image and the infrared image, capture complementary information of different modalities, and generate fusion features for target detection.

[0011] S4: Adopt a multi-scale detection head based on the FPN and PAN structures to perform target regression and classification, and enhance the detection accuracy of targets of different sizes.

[0012] Preferably, in S1, it further includes: removing the noise in the image to reduce the noise caused by the photosensitive device or transmission; standardizing the brightness and contrast of the image to ensure the consistency of image input; adjusting input images of different resolutions to a unified size for subsequent processing and calculation.

[0013] Preferably, S2 further includes: a local branch and a global branch, which are respectively responsible for learning the optical properties of objects and the illumination characteristics of images.

[0014] Further, the low-light enhancement method in S2 further includes: the local branch extracts local features of the image through a convolutional neural network, optimizes the pixel-level optical properties of the object, and adjusts and restores the object details through inverse mapping.

[0015] Further, the low-light enhancement method in S2 further includes: the global branch adopts an attention mechanism and a fully connected layer to improve the visibility of the image in a low-light environment by adjusting the overall brightness and contrast.

[0016] Preferably, step S3 further includes: using multiple self-attention mechanism fusion devices based on adaptive Transformer to perform modality fusion at different feature levels, enhancing the complementarity between modalities.

[0017] Preferably, step S4 further includes: adopting an adaptive anchor box generation method, through the K-means clustering algorithm, automatically generating adaptive anchor boxes according to the size distribution of the targets, improving the detection accuracy and robustness.

[0018] In a second aspect, an embodiment of the present invention further provides a multi-modal fusion remote sensing image target detection system for an unmanned aerial vehicle, including:

[0019] A low-light enhancement module: used to enhance low-light images, improving the brightness and details of the images.

[0020] A modality fusion module: used to perform feature fusion of the enhanced RGB image and the infrared image based on the Transformer self-attention mechanism.

[0021] A multi-scale target detection module: used to perform target regression and classification through multi-scale target detection technology.

[0022] The present invention has the following beneficial effects:

[0023] By proposing a novel multi-modal fusion remote sensing image target detection method, the present invention significantly improves the target detection performance of the unmanned aerial vehicle in low-light environments, and solves the problem of insufficient utilization of visible light image features in traditional methods under low-light conditions. The proposed multi-modal fusion detection framework can not only effectively enhance low-light images, but also incorporate the information of infrared images, thereby improving the detection accuracy and robustness.

[0024] By adopting a dual-branch low-light enhancement module based on the Retinex principle, this method can respectively extract the global and local features of the image. The local branch captures the pixel-level optical properties of the object itself, while the global branch learns the overall illumination characteristics of the image. Through this enhancement method, the present invention can effectively improve the quality of low-light images, making them more suitable for subsequent target detection tasks.

[0025] In addition, the present invention uses the self-attention mechanism based on Transformer for modality fusion, which can achieve deep interaction and information fusion within and between modalities, improving the effect and generalization ability of multi-modal image fusion. In particular, the self-attention mechanism optimizes the expression of the fused features by enhancing the dependence between different modalities, ensuring the accuracy of detecting targets in low-light environments. The design of this module enables the key information in the image to be fully extracted and optimized, thereby providing more distinguishable features for subsequent multi-scale target detection.

[0026] By integrating the multi-scale detection head module of FPN+PAN, the detection ability for targets of different sizes is further enhanced, ensuring the accurate recognition of small objects in complex environments. This multi-scale detection structure can not only improve the effect of target detection, but also reduce the consumption of computing resources, thus meeting the requirements of the drone system for real-time performance and light weight.

[0027] In summary, the present invention can not only significantly improve the target detection accuracy in low-light and complex environments, but also reduce the computational burden through lightweight design, adapting to the limited computing resources of drones. This makes the present invention have strong practicability and can be widely applied in the field of drone remote sensing, especially providing more accurate and stable target detection support in tasks such as power inspection, emergency rescue, and intelligent agriculture and forestry. Brief Description of the Drawings

[0028] Figure 1 It is a flowchart of the multi-modal low-light target detection of the present invention.

[0029] Figure 2 It is a flowchart of the low-light image enhancement of the present invention.

[0030] Figure 3 It is a flowchart of the multi-modal fusion of the present invention.

[0031] Figure 4 It is a neural network diagram of the multi-modal low-light target detection of the present invention Detailed Embodiments

[0032] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the flowcharts of the present invention. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0033] Combined Figure 1 , the present invention provides a multi-modal low-light detection method and system for drone remote sensing images, including the following steps:

[0034] Step S1, denoise, standardize and adjust the size of the input remote sensing image to ensure image quality and consistency, and stabilize subsequent processing steps:

[0035] The size of the image collected by the sensor is 1024×768. Among them, the image collected by the infrared camera has only a single channel, while the visible light image collected by the RGB camera has 3 channels. Wavelet transform denoising, standardization and linear interpolation size adjustment are performed on the infrared image and the visible light image respectively.

[0036] First, perform a discrete wavelet transform on the image to decompose the image into low-frequency and high-frequency components. For an RGB image, the low-frequency and high-frequency coefficients obtained are A R , H R , V R , D R (red channel), A G , H G , V G , D G (green channel), A B , H B , V B , D B (blue channel); for an infrared image, the coefficients A, H, V, D are obtained by decomposition. Apply soft threshold denoising to the high-frequency coefficients:

[0037] c′ j,k = sign(c j,k ) × max(|c j,k | - T, 0)

[0038] where c j,k is the high-frequency coefficient, T is the threshold, and sign(c j,k ) is the sign of the coefficient. Perform an inverse wavelet transform on the processed coefficients to reconstruct the denoised image.

[0039] I′ IR = IWT(A, H′, V′, D′)

[0040] I′ RGB = IWT(A R , H′ R , V′ R , D′ R , A G , H′ G , V′ G , D′ G , A B , H′ B , V′ B , D′ B )

[0041] where IWT represents the inverse wavelet transform, and I′ IR , I′ RGB represent the denoised infrared and visible light images respectively.

[0042] Normalize the denoised image and crop it to an appropriate size.

[0043] Step S2: Based on the Retinex principle, enhance the low-light image through a dual-branch network to improve the details and contrast of the image.

[0044] The reflection branch learns the optical properties Q of the object itself and is responsible for inverse mapping adjustment. The illumination branch learns the illumination property L, i.e., ISP parameters:

[0045] I enhanced = Q(L(I origin ))

[0046] where I enhanced represents the enhanced image features, and I origin represents the original low-light image information. Specifically, the illumination branch fits the inverse mapping process of the original image by the least squares method, and the reflection branch uses the attention mechanism to predict the global information of the image.

[0047] The low-light RGB image input to the reflection branch first expands the number of channels through a 3×3 convolutional kernel, and then encodes the position information through a 3×3 depth convolution to capture the spatial feature relationship of the input feature map. Subsequently, the feature map is successively convolved through channels and depth and concatenated with the original feature map after position encoding, and then input into the non-linear layer and channel convolution layer to achieve feature enhancement.

[0048] The image input to the illumination branch is first encoded through a 3×3 convolutional kernel to expand the channel dimension and reduce the resolution of the input features. Subsequently, after depth convolution position encoding, the features are input into the fully connected layer and projected into token vectors K and V. The two vectors and the preset vector Q form the attention mechanism operation, and finally, they are weighted through the fully connected, non-linear layer, and channel convolution to generate the global information of the image. The local and global information of the dual branches are concatenated and fused to generate the final enhanced image.

[0049] The reflection branch is responsible for learning the optical properties of the object and performing inverse mapping adjustment to restore the details of the object; the illumination branch is responsible for learning the global illumination characteristics of the image and optimizing the visibility of the image by adjusting the brightness and contrast of the image.

[0050] Step S3: Use a Transformer model based on the self-attention mechanism to perform feature interaction and fusion between the enhanced RGB image and the infrared image, capture complementary information of different modalities, and generate fusion features for object detection.

[0051] The specific process includes processing of input features, feature concatenation and position encoding, multi-head self-attention calculation, and generation of fusion features.

[0052] The input images are the enhanced RGB image and the infrared image respectively. Assume the enhanced RGB image is and the infrared image is where C represents the number of channels, and H and W represent the height and width of the image respectively.

[0053] To enable the Transformer to capture the spatial relationship information between input sequences, the flattened feature sequences I of RGB and infrared images R and I T are concatenated to obtain a fused sequence where 2HW represents the length of the concatenated sequence. Then, a learnable positional encoding I pos is added to the concatenated sequence to obtain the final input sequence. The role of this positional encoding is to help the Transformer retain the positional information of each feature position when processing the input features.

[0054] I = concat(I R + I T ) + I pos

[0055] After obtaining the fused feature sequence I, the multi-head self-attention mechanism of the Transformer is then used for feature interaction and fusion. First, the input sequence I is projected into query, key, and value vectors through a linear transformation:

[0056] Q = W Q ·I, K = W K ·I, V = W V ·I

[0057] where is the weight matrix, and D Q , D K , D V represent the dimensions of the query, key, and value vectors respectively. Next, the dot product of the query vector Q and the key vector K is calculated to obtain the correlation matrix A, and a scaling factor is used to prevent the numerical value from being too large:

[0058]

[0059] This correlation matrix A describes the dependency relationships between the various positions of the input features. Then, this matrix is used to perform a weighted sum of the value vector V to obtain the self-attention output:

[0060] Z = A·V

[0061] To capture more comprehensive dependency relationships, the multi-head self-attention mechanism is used. The calculation results of multiple attention heads are concatenated and then linearly transformed:

[0062] Z′ = concat(Z1, Z1, …, Z h )·W O

[0063] where h is the number of attention heads, is the output weight matrix.

[0064] After each self-attention calculation, use a residual connection to add the input I to the self-attention output Z':

[0065] O = MLP(Z' + I) + Z' + I

[0066] where MLP is a multi-layer perceptron module, including two fully connected layers and a non-linear activation function.

[0067] After obtaining the output feature O after self-attention calculation and residual connection, perform an inverse operation to restore it to the size of the original image.

[0068] F' R = reshape(O R ), F' T = reshape(O T )

[0069] Step S4: Use a multi-scale detection head based on the FPN and PAN structures for object regression and classification to enhance the detection accuracy of objects of different sizes.

[0070] The input is the multi-scale feature maps extracted by the backbone network, including the deep feature map (P5) and the shallow feature map (P3). The input feature maps are respectively: where C is the number of channels, and H and W are the height and width of each feature map respectively.

[0071] For the feature map P5, first process it through a convolutional layer to obtain a new feature map: Then perform an upsampling operation to make the spatial size of P5 match that of P4. Finally, concatenate the upsampled feature map with P4:

[0072]

[0073] where, represents the concatenation operation. Then, P'4 will continue to be processed through convolution to obtain a new feature map P''4. This process enhances the semantic information on P4.

[0074] Perform convolution on P3 and upsample to obtain a new feature map. Then, perform a concatenation operation on P''3 and P''4 to obtain an intermediate feature map P'''4, and perform convolution to obtain the final feature map:

[0075]

[0076] At this time, after being processed by the FPN and PAN structures, the obtained feature maps of each layer contain stronger semantic information and localization information, which are suitable for subsequent object regression and classification tasks.

[0077] Design a comprehensive loss function, including regression loss and classification loss. The regression loss uses the smooth L1 loss function, while the classification loss uses the cross-entropy loss.

[0078] Utilize the adaptive anchor box generation method, use the K-means algorithm to cluster the target box sizes, and calculate the anchor box sizes according to the clustering results. Finally, through the multi-scale fusion of the FPN and PAN structures and target regression and classification, generate the detection results of the targets.

[0079] A system for target detection in UAV remote sensing images, comprising:

[0080] A low-light enhancement module, where the reflection branch is responsible for learning the optical properties of objects and performing inverse mapping adjustment to restore the details of objects; the illumination branch is responsible for learning the global illumination characteristics of the image and optimizing the visibility of the image by adjusting the brightness and contrast of the image;

[0081] A modality fusion module, using multiple self-attention mechanism fusers based on adaptive Transformers to perform modality fusion at different feature levels to enhance the complementarity between modalities;

[0082] A multi-scale target detection module for target regression and classification according to the multi-scale detection head described in claim 5.

Claims

1. A method for detecting targets in UAV remote sensing images with multi-modal fusion, characterized in that, It includes the following steps: S1: Denoise, standardize, and adjust the size of the input remote sensing image to ensure image quality and consistency, and stabilize subsequent processing steps. S2: Based on the Retinex principle, enhance the low-light image through a dual-branch network to improve the details and contrast of the image. S3: Adopt a Transformer model based on the self-attention mechanism to perform feature interaction and fusion between the enhanced RGB image and the infrared image, capture complementary information of different modalities, and generate fusion features for object detection. S4: Adopt a multi-scale detection head based on the FPN and PAN structures for object regression and classification to enhance the detection accuracy of objects of different sizes.

2. The multi-modal fusion remote sensing image target detection method according to claim 1, wherein, The low-light enhancement module includes a local branch and a global branch, which are responsible for learning the optical properties of objects and the illumination characteristics of images respectively.

3. The multi-modal fusion remote sensing image target detection method according to claim 2, wherein, The local branch is responsible for learning the optical properties of objects and performing inverse mapping adjustment to restore the details of objects. The global branch is responsible for learning the global illumination characteristics of the image and optimizing the visibility of the image by adjusting the brightness and contrast of the image.

4. The multi-modal fusion remote sensing image target detection method according to claim 1, wherein, The modality fusion module uses multiple self-attention mechanism fusioners based on adaptive Transformer to perform modality fusion at different feature levels to enhance the complementarity between modalities.

5. The multimodal fusion remote sensing image target detection method according to claim 1, wherein, The multi-scale object detection module automatically generates adaptive anchor boxes according to different object sizes through an adaptive anchor box generation method, and enhances the detection accuracy through the FPN and PAN structures.

6. The multimodal fusion remote sensing image target detection method according to claim 5, wherein, The adaptive anchor box generation method uses the K-means algorithm to cluster the target box sizes and calculates the anchor box sizes according to the clustering results.

7. The multi-modal fusion remote sensing image target detection method according to claim 1, wherein, The method is applicable to object detection under low-light conditions, can effectively improve the detection accuracy, and has low computational overhead, making it suitable for real-time tasks.

8. A system for target detection in UAV remote sensing images, characterized in that, It includes: A low-light enhancement module for enhancing the low-light image according to the method described in claim 2; A modality fusion module for performing image feature fusion according to the adaptive Transformer mechanism described in claim 4; A multi-scale object detection module for performing object regression and classification according to the multi-scale detection head described in claim 5.

9. The system according to claim 8, wherein, The hardware platform of the system includes a multi-core processor and at least one GPU, supporting end-to-end training and inference tasks based on Pytorch.

10. The system according to claim 8, wherein, The experimental results of the system on the VisDrone and LLVIP datasets show that it can significantly improve the object detection accuracy and has low computational overhead, making it suitable for real-time drone tasks.

Images