A Method and Apparatus for Unmanned Aerial Vehicle (UAV) Target Detection Based on Infrared and Visible Image Fusion

Abstract

This invention provides a method and apparatus for UAV target detection based on infrared and visible light image fusion. The method includes: simultaneously acquiring images of the same scene in two modalities using sensors mounted on the UAV; extracting global illumination information from the visible light image to generate illumination weights; calculating weighting coefficients for the two modalities based on the illumination weights; weighting the visible light feature map and the infrared feature map using the corresponding weighting coefficients to obtain weighted features; determining the frequency domain difference between the weighted features of the two modalities; enhancing the weighted features based on the frequency domain difference to obtain enhanced features; concatenating the enhanced features of the two modalities and inputting them into a global branch and a local branch respectively to obtain a global offset field and a local offset field, which are then superimposed to obtain a final offset field; implicitly aligning the infrared enhanced features using the final offset field; and concatenating the aligned enhanced features of the two modalities and inputting them into a target detection head to obtain the target detection result. This invention improves the accuracy and robustness of target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and apparatus for UAV target detection based on infrared and visible light image fusion. Background Technology

[0002] With the widespread application of unmanned aerial vehicles (UAVs) in fields such as surveillance, traffic management, and emergency rescue, UAV-based target detection technology has gradually become an important research direction in computer vision. Traditional UAV vision systems mainly rely on visible light (RGB) images for detection and recognition. This mode has rich color and texture information and can provide high detection accuracy in well-lit environments. However, the performance of visible light images degrades significantly in complex environments such as low light, nighttime, backlight, smoke, or haze, leading to reduced detection accuracy or even target loss.

[0003] To overcome this problem, researchers introduced infrared (IR) imaging technology. Infrared images can stably reflect the thermal radiation characteristics of targets in low-light or no-light environments, unaffected by visible light illumination. Infrared and visible light images are significantly complementary: visible light images provide fine color and texture information; infrared images provide stable structural and contour features. Therefore, multimodal fusion of infrared and visible light images can significantly improve target detection.

[0004] However, existing infrared and visible light image fusion detection technologies still suffer from the following major problems: 1. Weak spatial alignment: Due to differences in the installation position, viewing angle, and imaging delay of infrared and visible light sensors, spatial displacement (parallax) often exists between the images of the same scene acquired by the two sensors. This asynchronous imaging leads to offset or scale mismatch of the target area in the two modal images. Existing methods mostly employ pixel-level or ROI-level explicit registration methods, such as registration algorithms based on optical flow or feature point matching. However, these methods are computationally complex, sensitive to geometric deformation, and struggle to maintain stable real-time performance in scenarios involving UAV movement or rapid target changes. 2. Modal information imbalance: Infrared and visible light images exhibit significant differences in their response to targets under different lighting conditions. For example, in strong daylight environments, visible light features dominate; while at night or in low light conditions, infrared features are more discriminative. Existing fusion algorithms mostly use fixed weights or simple stitching for feature fusion, failing to dynamically adjust modal contributions according to lighting conditions. This results in a "over-reliance on a single modality" problem in the fusion results, thus affecting detection accuracy. 3. Modal heterogeneity caused by illumination variations: In practical UAV applications, illumination distribution is often uneven, with differences in brightness, shadows, or localized light pollution in different areas. Existing methods typically ignore the frequency response changes caused by illumination differences, causing fused features to fail in low-light regions. 4. Separation of fusion and alignment: Most existing methods separate modal alignment and feature fusion into two independent stages: explicit alignment followed by feature fusion. This separate strategy increases computational complexity, and a large initial registration error directly impacts subsequent fusion and detection performance.

[0005] To address the aforementioned issues, academia and industry have proposed various improvement methods. For example, they employ attention-based modal weighted networks to enhance feature selectivity; introduce Transformer-based cross-modal feature interaction mechanisms; and utilize explicit geometric alignment (such as optical flow estimation and affine transformation) to alleviate parallax. However, these methods still have two limitations: (1) they lack a systematic consideration of adaptability to lighting conditions and cannot dynamically adjust the modal fusion ratio under day and night or complex lighting environments. (2) most of them rely on explicit geometric alignment mechanisms, which have high computational cost and poor real-time performance, making them unsuitable for the lightweight deployment requirements of UAVs. Summary of the Invention

[0006] This invention provides a method and apparatus for UAV target detection based on infrared and visible light image fusion, addressing the problems of weak alignment and illumination imbalance between infrared and visible light images in existing multimodal UAV target detection methods. This invention achieves UAV target detection through illumination-guided implicit alignment of infrared and visible light images, balancing illumination adaptability with implicit feature alignment capabilities, enabling high-precision, low-latency multimodal UAV target detection in complex environments. This method is primarily applied to multimodal image processing and target detection in complex lighting environments and can be widely used in scenarios such as UAV surveillance, traffic monitoring, security patrol, autonomous driving, and emergency rescue.

[0007] This invention provides a method for UAV target detection based on infrared and visible light image fusion, comprising:

[0008] The UAV simultaneously acquires images of the same scene in two modalities using both infrared and visible light imaging sensors. Global illumination information is extracted from the visible light images, and illumination weights are generated based on the global illumination information. The weighting coefficients for the two modalities are then calculated based on the illumination weights.

[0009] Visible light feature maps are extracted from the visible light image, and infrared feature maps are extracted from the infrared image. Weighted features are obtained by weighting the visible light feature maps and infrared feature maps with corresponding weighting coefficients.

[0010] The frequency domain difference between the weighted features of the two modes is determined, and the weighted features of the two modes are enhanced based on the frequency domain difference to obtain the enhanced features of the two modes;

[0011] The enhanced features of the two modes are concatenated and then input into the global branch and local branch respectively to obtain the global offset field and the local offset field. The global offset field and the local offset field are superimposed to obtain the final offset field. The final offset field is used to implicitly align the enhanced features of the infrared feature map.

[0012] The enhanced features of the two aligned modalities are concatenated and then input into the target detection head to obtain multi-class target detection results from the perspective of the UAV.

[0013] According to the present invention, a UAV target detection method based on infrared and visible light image fusion extracts global illumination information from a visible light image and generates two modal illumination weights based on the global illumination information, including:

[0014] After converting the visible light image to the Lab color space, the luminance channel L is extracted, and the color adversarial channel features RG and B-(R+G) / 2 are extracted from the visible light image.

[0015] The brightness channel features and color adversarial channel features are used as global illumination information input into the convolutional network to obtain the illumination weights output by the convolutional network.

[0016] According to the present invention, a UAV target detection method based on infrared and visible light image fusion calculates the weighting coefficients of the two modes according to the illumination weights, including:

[0017] Based on the illumination weights, calculate the weighting coefficients of the visible light feature map;

[0018] The weighting coefficients of the infrared feature map are calculated based on the weighting coefficients of the visible light feature map.

[0019] According to the present invention, a UAV target detection method based on infrared and visible light image fusion calculates the weighting coefficients of the two modes based on the illumination weights using the following formula:

[0020]

[0021] = 1 -

[0022] in, These are the weighting coefficients for the visible light feature map. These are the weighting coefficients for the infrared feature map. β is the illumination weight, and β is the adjustment coefficient.

[0023] According to the present invention, a method for UAV target detection based on infrared and visible light image fusion determines the frequency domain difference between the weighted features of the two modalities, including:

[0024] The spectral features are obtained by performing two-dimensional discrete cosine transform on the weighted features of the two modes respectively;

[0025] After fusing the weighted features of the two modes, a channel attention mechanism is applied to obtain global features, and modulation coefficients of the two modes are generated based on the global features.

[0026] The spectral characteristics of the two modes are modulated using the corresponding modulation coefficients to obtain the modulated characteristics;

[0027] The frequency domain features are obtained by performing a two-dimensional discrete cosine transform on the features modulated by the two modes.

[0028] The difference between the frequency domain features of the visible light image and the frequency domain features of the infrared image is taken as the first frequency domain difference, and the difference between the frequency domain features of the infrared image and the frequency domain features of the visible light image is taken as the second frequency domain difference.

[0029] According to the present invention, a UAV target detection method based on infrared and visible light image fusion is provided, which enhances the weighted features of the two modes according to the frequency domain differences to obtain enhanced features of the two modes, including:

[0030] The first frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first query; the second frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first key and the first value.

[0031] Based on the first query, the first key, and the first value, a first cross-modal attention matrix is ​​calculated, and the enhanced features of the visible light image are obtained by matrix multiplication based on the first cross-modal attention matrix.

[0032] The second frequency domain difference is downsampled and convolutionally reduced to obtain the second query, and the first frequency domain difference is downsampled and convolutionally reduced to obtain the second key and the second value.

[0033] Based on the second query, the second key, and the second value, a cross-modal attention matrix is ​​calculated, and the enhanced features of the infrared image are obtained by matrix multiplication based on the second cross-modal attention matrix.

[0034] According to the present invention, a UAV target detection method based on infrared and visible light image fusion is provided, wherein the global branch is a multilayer perceptron and the local branch is a convolutional network.

[0035] The present invention also provides a drone target detection device based on infrared and visible light image fusion, comprising:

[0036] The illumination sensing module is used to simultaneously acquire images of the same scene in two modalities using an infrared imaging sensor and a visible light imaging sensor mounted on the UAV, extract global illumination information from the visible light image and generate illumination weights based on the global illumination information, and calculate the weighting coefficients of the two modalities based on the illumination weights.

[0037] The feature weighting module is used to extract visible light feature maps from the visible light image, extract infrared feature maps from the infrared image, and weight the visible light feature maps and infrared feature maps respectively using corresponding weighting coefficients to obtain weighted features.

[0038] A frequency domain enhancement module is used to determine the frequency domain difference between the weighted features of the two modes, and enhance the weighted features of the two modes according to the frequency domain difference to obtain the enhanced features of the two modes;

[0039] The implicit alignment module is used to concatenate the enhanced features of the two modes and input them into the global branch and local branch respectively to obtain the global offset field and the local offset field. The global offset field and the local offset field are superimposed to obtain the final offset field. The final offset field is used to implicitly align the enhanced features of the infrared feature map.

[0040] The target detection module is used to concatenate the enhanced features of the two aligned modalities and input them into the target detection head to obtain multi-class target detection results from the perspective of the UAV.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the UAV target detection method based on infrared and visible light image fusion as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the UAV target detection method based on infrared and visible light image fusion as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the UAV target detection method based on infrared and visible light image fusion as described above.

[0044] The present invention provides a UAV target detection method and device based on infrared and visible light image fusion. By introducing a light perception mechanism, frequency domain modulation, cross-modal difference enhancement and implicit alignment dynamic fusion technology, it achieves light-adaptive modal fusion and high-precision target detection. It effectively solves the registration and feature imbalance problems of infrared and visible light images in complex lighting environments, and significantly improves the accuracy and robustness of UAV multimodal target detection. It can achieve highly robust multimodal fusion UAV target detection in complex lighting and weak alignment scenarios, and can be widely used in multimodal visual detection scenarios such as UAV monitoring, autonomous driving, and security inspection. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the UAV target detection method based on infrared and visible light image fusion provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the model architecture of the UAV target detection method based on infrared and visible light image fusion provided by the present invention;

[0048] Figure 3 This is a schematic diagram of the illumination weights in the UAV target detection method based on infrared and visible light image fusion provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of the UAV target detection device based on infrared and visible light image fusion provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined Figure 1 This invention describes a method for unmanned aerial vehicle (UAV) target detection based on infrared and visible light image fusion, comprising:

[0052] Step 101: Simultaneously acquire images of the same scene in two modalities using the infrared imaging sensor and the visible light imaging sensor mounted on the UAV; extract global illumination information from the visible light image and generate illumination weights based on the global illumination information; calculate the weighting coefficients of the two modalities based on the illumination weights.

[0053] Step 102: Extract visible light feature map from the visible light image, extract infrared feature map from the infrared image, and weight the visible light feature map and infrared feature map respectively using corresponding weighting coefficients to obtain weighted features;

[0054] Step 103: Determine the frequency domain difference between the weighted features of the two modes, and enhance the weighted features of the two modes according to the frequency domain difference to obtain the enhanced features of the two modes;

[0055] Step 104: After splicing the enhanced features of the two modes, input them into the global branch and local branch respectively to obtain the global offset field and the local offset field. Superimpose the global offset field and the local offset field to obtain the final offset field. Use the final offset field to implicitly align the enhanced features of the infrared feature map.

[0056] Step 105: After aligning the enhanced features of the two modalities, the data is concatenated and input into the target detection head to obtain multi-class target detection results from the UAV's perspective.

[0057] This embodiment provides an illumination-guided implicit alignment method for UAV target detection in infrared and visible light images, which can run in real time on the computing platform carried by the UAV. The model architecture is as follows: Figure 2 As shown, the specific steps include:

[0058] First, multimodal image acquisition and preprocessing. Infrared images of the same scene are acquired simultaneously using infrared and visible light sensors mounted on the drone. With visible light images .

[0059] Images from the two modalities may differ in viewpoint, pose, and resolution during acquisition, necessitating standardized preprocessing. Preprocessing includes image normalization and resizing, scaling both infrared and visible light images to the same size (e.g., standard resolutions like 640×512 or 1024×1024) and performing pixel mean-variance normalization to limit their values ​​to [0,1]. Furthermore, data augmentation operations such as brightness perturbation, modality loss, random cropping, flipping, and scaling can be performed on the images from both modalities for training purposes, improving the model's robustness under different lighting and viewpoint conditions.

[0060] Second, the generation of the illumination-aware weight map. The Opponent-aware LightIllumination Module (OLIM) extracts global illumination information through color adversarial features and the luminance channel to generate illumination weights, such as... Figure 3 As shown.

[0061] Third, illumination-guided frequency domain modulation. This involves modifying the infrared feature map... With visible light feature map Input the illumination-guided frequency domain modulation module (IFMM) and import the generated illumination weight map. The module calculates weighting coefficients to perform weighted fusion of infrared and visible light features. It adaptively adjusts the fusion ratio of the two modal features based on illumination intensity, mitigating modal imbalances caused by changes in illumination.

[0062] Fourth, frequency domain difference enhancement. The weighted features of the two modes are input into the Frequency-Guided Cross-Modality Differential Enhancement Module (FG-CMDEM). Frequency domain difference enhancement is performed on the weighted features of the two modes. By calculating the cross-modal frequency difference, structural detail information is extracted, and the target features in weakly aligned and low-contrast regions are enhanced.

[0063] Fifth, implicit alignment dynamic fusion. The enhanced features of the two modalities are input into the implicit alignment dynamic fusion module (IADDF module). The spatial offset field is predicted by a global multilayer perceptron (MLP) and local convolutional branches to implicitly align the infrared features and generate dynamic convolutional kernels to achieve position-adaptive feature fusion.

[0064] Sixth, target detection and output. This involves fusing features. Input a detection head network (such as YOLOv8), output the target category and bounding box coordinates, and complete infrared and visible light multimodal UAV target detection.

[0065] The detection head can adopt a YOLOv8-based object detection structure, and the loss function used during training includes a weighted sum of class loss (such as cross-entropy or Focal Loss), regression loss (such as CIoU or GIoU), and confidence loss. Training optimization uses optimizers such as SGD or AdamW, with a learning rate ranging from 1e-5 to 1e-3 and can employ cosine annealing or multi-stage descent strategies.

[0066] It also includes training-phase augmentations: data augmentation such as brightness perturbation, Gaussian noise, mode loss (randomly discarding infrared or visible light), and random affine transformations to improve robustness.

[0067] All modules (OLIM, IFMM, FG-CMDEM, IADDF) are trained as a single end-to-end differentiable network during training. To ensure stable training, a phased training strategy can be adopted: first train the basic detection backbone and detection head, then add IFMM and FG-CMDEM in stages, and finally fine-tune IADDF.

[0068] This embodiment introduces a light-sensing mechanism, frequency domain modulation, cross-modal difference enhancement, and implicit alignment dynamic fusion technology to achieve light-adaptive modal fusion and high-precision target detection. It effectively solves the registration and feature imbalance problems of infrared and visible light images in complex lighting environments, significantly improving the accuracy and robustness of UAV multimodal target detection. It can achieve highly robust multimodal fusion UAV target detection in complex lighting and weak alignment scenarios, and can be widely used in multimodal visual inspection scenarios such as UAV monitoring, autonomous driving, and security patrol.

[0069] Based on the above embodiments, this embodiment extracts global illumination information from a visible light image and generates two modal illumination weights based on the global illumination information, including:

[0070] After converting the visible light image to the Lab color space, the luminance channel feature L is extracted, and the color adversarial channel features RG and B-(R+G) / 2 are extracted from the visible light image.

[0071] The brightness channel features and color adversarial channel features are used as global illumination information input into the convolutional network to obtain the illumination weights output by the convolutional network.

[0072] Visible light images Convert from RGB color space to Lab color space and extract luminance channel features. Simultaneously calculate the color adversarial channel features:

[0073]

[0074]

[0075] Will , and splicing to form a three-channel input This serves as input to the Light-Aware Module (OLIM) to generate a lighting weight map that reflects the spatial distribution of scene lighting.

[0076] The illumination sensing module is used to extract scene illumination feature information and generate an illumination weight map. .

[0077] The illumination sensing module consists of several levels of convolutional units, each level including a convolutional layer, a normalization layer, and a non-linear activation layer. The preferred structure is: three convolutional layers (3×3 kernel size, stride 2), with SiLU or ReLU activation functions; each layer is followed by a batch normalization layer and an activation function; the last layer uses a 1×1 convolution to compress channels and outputs an illumination weight map using a Sigmoid function; the OLIM network depth is 2 to 6 convolutional layers, and the illumination weight map... The spatial resolution is typically 1 / 4 or 1 / 8 of the input image, with a numerical range of [0,1].

[0078] The calculation process of the illumination sensing module is as follows:

[0079]

[0080] Where σ is the Sigmoid function, and the output is the illumination weight map. The pixel values ​​are limited to [0, 1].

[0081] Three-channel input The image is sequentially passed through several convolutional units (each unit being Conv→BN→SiLU or ReLU), downsampled to the desired feature map size; then, it is passed through a 1×1 convolution and activated by a Sigmoid function to output an illumination weight map. Its pixel value is limited to [0,1], and is used to characterize the local and global illumination distribution of the scene.

[0082] The illumination weight map reflects the local and global illumination intensity in the scene, and its output size is usually matched with the resolution of the input feature map (such as 1 / 4 or 1 / 8 of the original image size).

[0083] This illumination weighting map is used to adaptively adjust the fusion ratio of infrared and visible light features in the subsequent frequency domain modulation module. In high-brightness regions, It tends to enhance visible light feature contribution; in low-light or nighttime areas, This enhances the weight of infrared features, thereby dynamically balancing the modal contribution under strong or weak light conditions.

[0084] Based on the above embodiments, this embodiment calculates the weighting coefficients of the two modes according to the illumination weights, including:

[0085] Based on the illumination weights, calculate the weighting coefficients of the visible light feature map;

[0086] The weighting coefficients of the infrared feature map are calculated based on the weighting coefficients of the visible light feature map.

[0087] The illumination-guided frequency-domain modulation module is used to achieve illumination-adaptive feature fusion at the frequency domain level, thereby suppressing the impact of illumination variations on the consistency of multimodal features. This involves converting visible light feature maps... With infrared feature map The input light-guided frequency domain modulation module performs the following processing:

[0088] (1) Weight Calculation: Calculate the modal weighting coefficients based on the illumination weighting map:

[0089] = , = 1 - ;

[0090] (2) Feature weighting and convolution fusion: for two-modal features and Each with the corresponding , Perform channel broadcast multiplication and then insert the results into a grouped 1×1 convolution to obtain weighted features. , :

[0091]

[0092] in, This represents element-wise multiplication;

[0093] Will and Fusion characteristics are obtained through elemental addition. And apply channel attention (Squeeze-and-Excitation or a similar mechanism) to obtain a global representation. ;

[0094] (3) Frequency domain modulation: by Two sets of frequency domain modulation weights are generated by 1×1 convolution and then processed by SiLU or GELU. and ;

[0095] right and The spectral characteristics are obtained by applying two-dimensional discrete cosine transform (2D DCT) to local spatial locations. Then, through the generated frequency domain modulation weights , Weighting of different modal spectral channels yields the spatial characteristics after frequency domain modulation. , In the frequency domain, adaptive enhancement and reconstruction of illumination are performed using the following formula:

[0096]

[0097] in, IDCT is the inverse transform. represent , In coordinates The value on, represent In feature map The value on, Representative characteristics and coordinates on This represents element-wise multiplication of features.

[0098] Modulation characteristics of the output It achieves adaptive illumination correction in the frequency domain, which can effectively reduce modal response deviations caused by brightness changes.

[0099] 2D DCT or IDCT can be performed independently on each channel, and the application block size of DCT is... ∈{8,16,32}; When using block processing, either overlapping or non-overlapping strategies between blocks can be employed.

[0100] This module can dynamically adjust the response ratio of different modes in the frequency space according to the scene's illumination intensity.

[0101] This eliminates the modal imbalance problem under strong light, backlight, or dark light conditions.

[0102] Based on the above embodiments, this embodiment uses the following formula according to the illumination weight. The weighting coefficients for the two modes are calculated per pixel:

[0103]

[0104] = 1 -

[0105] in, These are the weighting coefficients for the visible light feature map. These are the weighting coefficients for the infrared feature map. β is the illumination weight, and β is the adjustment coefficient. The value of β ranges from 0.1 to 0.7, with 0.3 being preferred.

[0106] Based on the above embodiments, this embodiment determines the frequency domain difference between the weighted features of the two modes, including:

[0107] The spectral features are obtained by performing two-dimensional discrete cosine transform on the weighted features of the two modes respectively;

[0108] After fusing the weighted features of the two modes, a channel attention mechanism is applied to obtain global features, and modulation coefficients of the two modes are generated based on the global features.

[0109] The spectral characteristics of the two modes are modulated using the corresponding modulation coefficients to obtain the modulated characteristics;

[0110] The frequency domain features are obtained by performing a two-dimensional discrete cosine transform on the features modulated by the two modes.

[0111] The difference between the frequency domain features of the visible light image and the frequency domain features of the infrared image is taken as the first frequency domain difference, and the difference between the frequency domain features of the infrared image and the frequency domain features of the visible light image is taken as the second frequency domain difference.

[0112] Based on the above embodiments, this embodiment enhances the weighted features of the two modes according to the frequency domain differences to obtain enhanced features of the two modes, including:

[0113] The first frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first query; the second frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first key and the first value.

[0114] Based on the first query, the first key, and the first value, a first cross-modal attention matrix is ​​calculated, and the enhanced features of the visible light image are obtained by matrix multiplication based on the first cross-modal attention matrix.

[0115] The second frequency domain difference is downsampled and convolutionally reduced to obtain the second query, and the first frequency domain difference is downsampled and convolutionally reduced to obtain the second key and the second value.

[0116] Based on the second query, the second key, and the second value, a cross-modal attention matrix is ​​calculated, and the enhanced features of the infrared image are obtained by matrix multiplication based on the second cross-modal attention matrix.

[0117] The frequency domain difference enhancement module enhances feature representation capabilities through cross-modal frequency domain difference analysis, improving the perception of low-contrast and weak edge regions. It modulates features from two modes... , Input the frequency domain difference enhancement module and perform the following operations:

[0118] (1) To , Perform 2D DCT to obtain the frequency domain representation. and The formula is as follows:

[0119] ;

[0120] (2) Calculate the difference in the first frequency domain = - and the difference in the second frequency domain = - ;

[0121] (3) To Downsampling (downsampling factor s ∈ {2,4}) and dimensionality reduction through 1×1 convolution to obtain the query Q, key K, and value V matrix (dimensionality reduction channel). );

[0122] (4) Calculate the differential attention map and the attention mapping through the cross-modal attention mechanism. Multiply by V to obtain the difference enhancement feature. Let K be the dimension of the key. Then, the enhanced spatial features are added to the original spatial feature residuals via a feedforward network (FFN) and inverse DCT / reprojection to produce the enhanced spatial features: Finally, the results are mapped back to the spatial domain using inverse DCT.

[0123] Attention calculation can be performed using multi-head attention, with the number of heads h ∈ {1,2,4,8}; if it is a single head, then h=1.

[0124] This module extracts modal difference information at the frequency domain level, enhances the features of edges, contours and weakly aligned target regions, highlights complementary information between modalities through frequency domain differences and attention mechanisms, adaptively enhances the response of structural edges and weakly aligned target regions, and enables the fused features to have stronger structural perception capabilities. It can significantly enhance edge details and target salience in low light, haze or partial occlusion scenarios.

[0125] Based on the above embodiments, in this embodiment the global branch is a multilayer perceptron and the local branch is a convolutional network.

[0126] The Implicit Alignment Dynamic Fusion Module (IADDF) is used to achieve implicit spatial alignment and adaptive fusion of infrared and visible light features. It combines the enhanced features of the two modalities. The input implicit alignment dynamic blending module includes the following processing:

[0127] (1) Enhancement features of visible light and infrared light Obtained by splicing along the channels ;

[0128] (2) Global branch: After concatenating with a normalized coordinate map (each pixel coordinate (x,y) normalized to [-1,1]), the input is a two-layer MLP (layer width can be 256 or 512), and the output is a global displacement field. ∈ R^{2×H×W}, where H and W are the width and height of the image;

[0129] (3) Local branch: The local displacement field is output through 3×3 convolution (1 to 3 convolutional layers can be stacked). ∈ R^{2×H×W};

[0130] (4) Superimpose the two offset fields to obtain the final displacement field. = + Implicit spatial alignment for infrared features The value range is limited to ±32 pixels to ensure spatial stability.

[0131] (5) Use bilinear interpolation (Bilinear GridSample) for infrared Perform implicit resampling correction to obtain the aligned result. :

[0132] ;

[0133] (6) The aligned infrared features are concatenated with the visible light features and input into the dynamic convolution kernel generation unit. The dynamic convolution unit generates an adaptive convolution kernel based on the local features at each pixel position:

[0134] For each spatial location Predict the corresponding convolutional kernel weights using MLP nuclear size Preferably 3 or 5. Combine the convolution kernel with local patches. Perform pointwise convolution to obtain fused features ,Right now Finally, it is output after 1×1 convolution dimensionality reduction. .

[0135] When generating dynamic convolutional kernels, BatchNorm and SiLU / GELU activation are used for the weights, and to avoid overfitting, [further details can be added]. Apply L2 regularization or normalization (e.g., softmax normalization or normalization factor).

[0136] Displacement field The pixel values ​​are limited to a preset range (e.g., ±p pixels, where p ranges from 1 to 32) to ensure the stability of the geometric transformation.

[0137] This module achieves unsupervised implicit alignment of infrared-visible features through joint modeling of global and local offset predictions. It eliminates parallax and imaging offset problems without explicit geometric registration, resulting in more stable fusion results.

[0138] During the detection and output process, features will be fused. Input the detection head network. The detection head can be a convolutional detector based on the YOLOv8 architecture. Output results include the target class label, target bounding box coordinates, and detection confidence score.

[0139] During the training phase, the loss function can consist of the following parts:

[0140]

[0141] in, For classifying losses, For bounding box regression loss, This represents the confidence loss.

[0142] This embodiment uses the VEDAI and FLIR datasets to experimentally evaluate the illumination-guided implicit alignment method for infrared and visible light UAV target detection provided in this embodiment, in order to further illustrate the performance and effectiveness of this method.

[0143] The VEDAI dataset is a publicly available multimodal target detection dataset for drones, containing both visible light and infrared images. It covers various typical lighting environments, including daytime, nighttime, shadow, backlight, and strong light reflection, realistically reflecting the impact of complex lighting conditions on detection algorithms from the drone's perspective. The dataset contains approximately 1246 pairs of infrared and visible light images, labeled with categories including vehicles, pedestrians, bicycles, trucks, and other common traffic and surveillance targets, making it suitable for performance validation of multimodal detection algorithms.

[0144] The FLIR dataset contains paired visible light (RGB) and infrared (IR) images of daytime and nighttime scenes. It includes 5142 aligned RGB-IR pairs, with 4129 pairs used for training and 1013 pairs used for testing, focusing on the categories of "people," "vehicles," and "bicycles." The "dog" category was removed due to its rarity.

[0145] In the experiments, the method of this invention used the VEDAI and FLIR datasets as training and testing platforms, employing a unified data partitioning method, with approximately 70% of the data used for training, 20% for validation, and 10% for testing. The same hardware configuration and optimization parameters were used during training to ensure fairness with the comparison algorithm results.

[0146] Experimental results show that, under the same training parameters and hardware environment, this embodiment compares the performance of the method of the present invention and existing typical algorithms on the same dataset. The average detection accuracy (mAP) and AP50 of each method are shown. AP50 refers to the average accuracy when the Intersection over Union (IoU) threshold is 0.5, while mAP represents the average accuracy in the range of IoU threshold from 0.5 to 0.95. The metrics are shown in Table 1 and Table 2.

[0147] This invention achieves the best performance on the VEDAI dataset by comparing different methods in different modalities. The infrared-visible light fusion detection algorithm is significantly better than the single modality (visible light or infrared). Under the same modality type (visible light-infrared), this invention is better than ICAFSion, and it is 4.1% higher than ICAFSion in AP50 and 0.5% higher than ICAFSion in mAP.

[0148] This invention compares different methods to achieve the best performance on the FLIR dataset across different modalities. The infrared-visible light fusion detection algorithm significantly outperforms single-modality (visible light or infrared). This method achieves 85.0% on AP50, which is 4.9% and 17.2% higher than YOLOv5 single-modality (visible light or infrared), respectively. Under the same modal type (visible light-infrared), my invention outperforms UniRGB-IR, exceeding UniRGB-IR by 1.1% on AP50 and by 6% on mAP.

[0149] In summary, the experimental results based on the VEDAI and FLIR datasets show that the method of this invention has good generalization ability and robustness in complex lighting environments, and can achieve significant improvements in accuracy, alignment stability and real-time performance compared with existing infrared-visible light fusion detection algorithms.

[0150] Table 1. Comparison results of the present invention and different methods on the VEDAI dataset under different modalities.

[0151]

[0152] Table 2. Comparison results of the present invention and different methods on the FLIR dataset under different modalities.

[0153]

[0154] The UAV target detection device based on infrared and visible light image fusion provided by the present invention will be described below. The UAV target detection device based on infrared and visible light image fusion described below can be referred to in correspondence with the UAV target detection method based on infrared and visible light image fusion described above.

[0155] like Figure 4 As shown, the device includes a light sensing module 601, a feature weighting module 602, a frequency domain enhancement module 603, an implicit alignment module 604, and a target detection module 605, wherein:

[0156] The illumination sensing module 601 is used to simultaneously acquire images of the same scene in two modalities using an infrared imaging sensor and a visible light imaging sensor mounted on the UAV, extract global illumination information from the visible light image and generate illumination weights based on the global illumination information, and calculate the weighting coefficients of the two modalities based on the illumination weights.

[0157] The feature weighting module 602 is used to extract a visible light feature map from the visible light image, extract an infrared feature map from the infrared image, and weight the visible light feature map and the infrared feature map respectively using corresponding weighting coefficients to obtain weighted features;

[0158] The frequency domain enhancement module 603 is used to determine the frequency domain difference between the weighted features of the two modes, and enhance the weighted features of the two modes according to the frequency domain difference to obtain the enhanced features of the two modes;

[0159] The implicit alignment module 604 is used to stitch together the enhanced features of the two modes and input them into the global branch and local branch respectively to obtain the global offset field and the local offset field. The global offset field and the local offset field are superimposed to obtain the final offset field. The final offset field is used to implicitly align the enhanced features of the infrared feature map.

[0160] The target detection module 605 is used to stitch together the enhanced features of the two aligned modalities and input them into the target detection head to obtain multi-class target detection results from the perspective of the UAV.

[0161] This embodiment introduces a light-sensing mechanism, frequency domain modulation, cross-modal difference enhancement, and implicit alignment dynamic fusion technology to achieve light-adaptive modal fusion and high-precision target detection. It effectively solves the registration and feature imbalance problems of infrared and visible light images in complex lighting environments, significantly improving the accuracy and robustness of UAV multimodal target detection. It can achieve highly robust multimodal fusion UAV target detection in complex lighting and weak alignment scenarios, and can be widely used in multimodal visual inspection scenarios such as UAV monitoring, autonomous driving, and security patrol.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for UAV target detection based on infrared and visible light image fusion, characterized in that, include: The UAV simultaneously acquires images of the same scene in two modalities using both infrared and visible light imaging sensors. Global illumination information is extracted from the visible light images, and illumination weights are generated based on the global illumination information. The weighting coefficients for the two modalities are then calculated based on the illumination weights. Visible light feature maps are extracted from the visible light image, and infrared feature maps are extracted from the infrared image. Weighted features are obtained by weighting the visible light feature maps and infrared feature maps with corresponding weighting coefficients. The frequency domain difference between the weighted features of the two modes is determined, and the weighted features of the two modes are enhanced based on the frequency domain difference to obtain the enhanced features of the two modes; The enhanced features of the two modes are concatenated and then input into the global branch and local branch respectively to obtain the global offset field and the local offset field. The global offset field and the local offset field are superimposed to obtain the final offset field. The final offset field is used to implicitly align the enhanced features of the infrared feature map. The enhanced features of the two aligned modalities are concatenated and then input into the target detection head to obtain multi-class target detection results from the perspective of the UAV. Extracting global illumination information from a visible light image and generating illumination weights based on the global illumination information includes: After converting the visible light image to the Lab color space, the luminance channel L is extracted, and the color adversarial channel features RG and B-(R+G) / 2 are extracted from the visible light image. The brightness channel features and color adversarial channel features are used as global illumination information input into the convolutional network to obtain the illumination weights output by the convolutional network. Determining the frequency domain difference between the weighted features of the two modes includes: The spectral features are obtained by performing two-dimensional discrete cosine transform on the weighted features of the two modes respectively; After fusing the weighted features of the two modes, a channel attention mechanism is applied to obtain global features, and modulation coefficients of the two modes are generated based on the global features. The spectral characteristics of the two modes are modulated using the corresponding modulation coefficients to obtain the modulated characteristics; The frequency domain features are obtained by performing a two-dimensional discrete cosine transform on the features modulated by the two modes. The difference between the frequency domain features of the visible light image and the frequency domain features of the infrared image is taken as the first frequency domain difference, and the difference between the frequency domain features of the infrared image and the frequency domain features of the visible light image is taken as the second frequency domain difference. The enhanced features of the two modes are obtained by enhancing the weighted features of the two modes based on the frequency domain differences, including: The first frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first query; the second frequency domain difference is downsampled and dimensionality reduced by convolution, and then used as the first key and the first value. Based on the first query, the first key, and the first value, a first cross-modal attention matrix is ​​calculated, and the enhanced features of the visible light image are obtained by matrix multiplication based on the first cross-modal attention matrix. The second frequency domain difference is downsampled and convolutionally reduced to obtain the second query, and the first frequency domain difference is downsampled and convolutionally reduced to obtain the second key and the second value. Based on the second query, the second key, and the second value, a cross-modal attention matrix is ​​calculated, and the enhanced features of the infrared image are obtained by matrix multiplication based on the second cross-modal attention matrix.

2. The UAV target detection method based on infrared and visible light image fusion according to claim 1, characterized in that, The weighting coefficients for the two modes are calculated based on the illumination weights, including: Based on the illumination weights, calculate the weighting coefficients of the visible light feature map; The weighting coefficients of the infrared feature map are calculated based on the weighting coefficients of the visible light feature map.

3. The UAV target detection method based on infrared and visible light image fusion according to claim 1, characterized in that, The weighting coefficients for the two modes are calculated using the following formula based on the illumination weights: ； = 1 - ； in, These are the weighting coefficients for the visible light feature map. These are the weighting coefficients for the infrared feature map. β is the illumination weight, and β is the adjustment coefficient.

4. The UAV target detection method based on infrared and visible light image fusion according to claim 1, characterized in that, The global branch is a multilayer perceptron, and the local branch is a convolutional network.

5. A UAV target detection device based on infrared and visible light image fusion, characterized in that, The UAV target detection method based on infrared and visible light image fusion as described in any one of claims 1-4 includes: The illumination sensing module is used to simultaneously acquire images of the same scene in two modalities using an infrared imaging sensor and a visible light imaging sensor mounted on the UAV, extract global illumination information from the visible light image and generate illumination weights based on the global illumination information, and calculate the weighting coefficients of the two modalities based on the illumination weights. The feature weighting module is used to extract visible light feature maps from the visible light image, extract infrared feature maps from the infrared image, and weight the visible light feature maps and infrared feature maps respectively using corresponding weighting coefficients to obtain weighted features. A frequency domain enhancement module is used to determine the frequency domain difference between the weighted features of the two modes, and enhance the weighted features of the two modes according to the frequency domain difference to obtain the enhanced features of the two modes; The implicit alignment module is used to concatenate the enhanced features of the two modes and input them into the global branch and local branch respectively to obtain the global offset field and the local offset field. The global offset field and the local offset field are superimposed to obtain the final offset field. The final offset field is used to implicitly align the enhanced features of the infrared feature map. The target detection module is used to concatenate the enhanced features of the two aligned modalities and input them into the target detection head to obtain multi-class target detection results from the perspective of the UAV. The light sensing module is specifically used for: After converting the visible light image to the Lab color space, the luminance channel L is extracted, and the color adversarial channel features RG and B-(R+G) / 2 are extracted from the visible light image. The brightness channel features and color adversarial channel features are used as global illumination information input into the convolutional network to obtain the illumination weights output by the convolutional network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the UAV target detection method based on infrared and visible light image fusion as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV target detection method based on infrared and visible light image fusion as described in any one of claims 1 to 4.

Images