Unmanned aerial vehicle small target detection method and system based on bimodal image fusion

Abstract

The invention discloses an unmanned aerial vehicle small target detection method and system based on bimodal image fusion. The target detection method comprises the steps of obtaining a to-be-detected unmanned aerial vehicle image; the to-be-detected unmanned aerial vehicle image is input to a target detection model, a detection result is obtained, and the target detection model comprises a backbone network module, a neck network module, a detection module and an output module; the backbone network module is used for acquiring feature maps of different scales; the neck network module is used for performing different-level feature fusion on the feature map to obtain a fused feature map; the detection module is used for detecting the fused feature map to obtain a detection result; and the output module is used for outputting the detection result. The problems of low detection precision and high omission ratio caused by small target size and large modal information difference in small target detection of the unmanned aerial vehicle are solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method and system for detecting small targets of unmanned aerial vehicles based on dual-modal image fusion. Background Art

[0002] In recent years, unmanned aerial vehicle (UAV) technology has been widely used in fields such as military surveillance, disaster assessment, and agricultural monitoring. However, due to the characteristics of UAV images, such as the large number of small targets, complex backgrounds, and large variations in target scale, traditional single-modal target detection methods are prone to low detection accuracy and high missed detection rates in complex scenes.

[0003] Infrared images are sensitive to temperature differences and are suitable for target capture in low-light conditions such as at night and in fog and haze. They have the advantage of strong anti-interference ability, but their resolution is low and details are insufficient. Visible light images can provide rich detailed information under sufficient lighting conditions, but their performance decreases significantly in low-light or nighttime conditions. By fusing infrared and visible light images, we can solve the problems of high missed detection rate and high false detection rate in target detection under complex lighting conditions. However, most existing infrared and visible light fusion detection methods focus on simple feature splicing or early fusion, and do not fully consider the complementarity of modal features and the ability to express information. In particular, there is still much room for improvement in the performance of small target detection tasks. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a UAV small target detection method and system based on dual-modal image fusion, which can achieve efficient fusion of infrared and visible light images and comprehensive improvement of detection performance.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention provides a method for detecting small targets of unmanned aerial vehicles based on dual-modal image fusion, comprising:

[0007] Obtain images of the drone to be tested;

[0008] Input the image of the drone to be tested into a target detection model to obtain a detection result, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module;

[0009] The backbone network module is used to obtain feature maps of different scales;

[0010] The neck network module is used to fuse features at different levels on the feature map to obtain a fused feature map;

[0011] The detection module is used to detect the fused feature map and obtain the detection result;

[0012] The output module is used to output the detection result.

[0013] Optionally, the image of the drone to be tested includes: a visible light image and an infrared image.

[0014] Optionally, the backbone network module includes: a convolution block, a feature extraction unit, a lightweight downsampling unit, a visible light infrared fusion unit and a splicing unit;

[0015] The convolution block is used to extract local features of the visible light image and the infrared image, and perform normalization to obtain modal features of the visible light image and the infrared image;

[0016] The feature extraction unit is configured to obtain detailed information of the modal features by using a multi-scale convolution operation;

[0017] The lightweight downsampling unit is used to reduce the resolution of the modal feature;

[0018] The visible light infrared fusion unit is used to process the modal features using an adaptive weight allocation mechanism and a channel reorganization operation;

[0019] The splicing unit is used to fuse and splice the processed modal features.

[0020] Optionally, a method for obtaining detailed information of the modal features by using a multi-scale convolution operation is as follows:

[0021]

[0022] Among them, IB k×k represents the output of the inverse bottleneck layer with a kernel size of k×k, Y i-1 represents the output of the previous branch, X i is the i-th branch, Y i is the output of the i-th branch.

[0023] Optionally, processing the modal features using an adaptive weight allocation mechanism and a channel reorganization operation includes:

[0024] Performing a global average pooling operation on the modal features to obtain global information;

[0025] Processing the global information using a fully connected layer to obtain a weighted coefficient of the channel;

[0026] Obtaining a weighted feature map according to the weighting coefficient;

[0027] Split the weighted feature map along the channel dimension to obtain the first channel and the second channel;

[0028] The first channel and the second channel are interchanged and spliced ​​to obtain processed modal features.

[0029] The present invention also provides a UAV small target detection system based on dual-modal image fusion, comprising: an image acquisition subsystem and an image detection subsystem;

[0030] The image acquisition subsystem is used to acquire images of the drone to be tested;

[0031] The image detection subsystem is used to input the image of the drone to be tested into a target detection model to obtain a detection result, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module;

[0032] The backbone network module is used to obtain feature maps of different scales;

[0033] The neck network module is used to fuse features at different levels on the feature map to obtain a fused feature map;

[0034] The detection module is used to detect the fused feature map and obtain the detection result;

[0035] The output module is used to output the detection result.

[0036] Optionally, the backbone network module includes: a convolution block, a feature extraction unit, a lightweight downsampling unit, a visible light infrared fusion unit and a splicing unit;

[0037] The convolution block is used to extract local features of the visible light image and the infrared image, and perform normalization to obtain modal features of the visible light image and the infrared image;

[0038] The feature extraction unit is configured to obtain detailed information of the modal features by using a multi-scale convolution operation;

[0039] The lightweight downsampling unit is used to reduce the resolution of the modal feature;

[0040] The visible light infrared fusion unit is used to process the modal features using an adaptive weight allocation mechanism and a channel reorganization operation;

[0041] The splicing unit is used to fuse and splice the processed modal features.

[0042] Optionally, processing the modal features using an adaptive weight allocation mechanism and a channel reorganization operation includes:

[0043] Performing a global average pooling operation on the modal features to obtain global information;

[0044] Processing the global information using a fully connected layer to obtain a weighted coefficient of the channel;

[0045] Obtaining a weighted feature map according to the weighting coefficient;

[0046] Split the weighted feature map along the channel dimension to obtain the first channel and the second channel;

[0047] The first channel and the second channel are interchanged and spliced ​​to obtain processed modal features.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] This invention addresses the issues of low detection accuracy and high missed detection rates in small drone target detection, often caused by the small size of the target and significant differences in modal information. The combined design of the C2f_MS, ADown, and VIFM modules effectively improves the quality of feature extraction and modal fusion; the BiFPN module further enhances the multi-scale feature fusion capabilities. While maintaining high accuracy, this invention reduces the number of model parameters, providing an efficient and lightweight solution for drone target detection tasks. It has broad application value in fields such as disaster assessment and agricultural monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0051] Figure 1 This is a flow chart of a method for detecting small targets in a drone based on dual-modal image fusion according to an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of the structure of the target detection model according to an embodiment of the present invention;

[0053] Figure 3 4 is a schematic structural diagram of a VIFM unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] This embodiment proposes a method for detecting small targets of drones based on dual-modal image fusion, which specifically includes: acquiring an image of the drone to be detected;

[0057] Input the image of the drone to be tested into the target detection model to obtain the detection results, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module;

[0058] Backbone network module, used to obtain feature maps of different scales;

[0059] The neck network module is used to fuse features at different levels of the feature map to obtain the fused feature map;

[0060] The detection module is used to detect the fused feature map and obtain the detection results;

[0061] Output module, used to output detection results.

[0062] Specifically, such as Figure 1 As shown, the method includes: acquiring visible light and infrared images from drone images, designing the backbone network of YOLOv8 into a dual-branch network, and inputting the acquired images into the two branches respectively.

[0063] An efficient feature extraction module and downsampling module are introduced into the backbone network of YOLOv8 to obtain feature maps of different scales, and the feature fusion module with adaptive weights is used to fuse the feature maps.

[0064] The fusion results are extracted separately, spliced, and fed into the neck network. A bidirectional feature pyramid network is used to efficiently fuse shallow graphical features with deep semantic features, resulting in fused feature maps of different scales. By constructing a bimodal image fusion model, this paper effectively improves the low detection accuracy and high missed detection rate faced by drone target detection due to small target size and complex scenes, laying a good foundation for the widespread application of drones.

[0065] The fused feature maps of different scales are input into the detector for detection, and the detection results are output.

[0066] Furthermore, the drone images to be tested include: visible light images and infrared images.

[0067] Furthermore, the backbone network module includes: convolution block, feature extraction unit, lightweight downsampling unit (downsampling module ADown), visible light infrared fusion unit (VIFM module) and splicing unit;

[0068] Convolutional block, used to extract local features of visible light images and infrared images, and perform normalization to obtain modal features of visible light images and infrared images;

[0069] A feature extraction unit is used to obtain detailed information of modal features using multi-scale convolution operations;

[0070] Lightweight downsampling unit, used to reduce the resolution of modal features;

[0071] A visible-light infrared fusion unit for processing modal features using an adaptive weight allocation mechanism and channel recombination operations;

[0072] The splicing unit is used to fuse and splice the processed modal features.

[0073] Specifically, we build a dual-branch backbone network based on YOLOv8 and introduce related image processing methods. The methods for processing visible light and infrared images include:

[0074] Two-dimensional convolution blocks are used to extract local features and perform normalization and introduce nonlinear capabilities.

[0075] The efficient feature extraction module C2f_MS and downsampling module ADown are used to obtain complex features in the image.

[0076] The feature information in the infrared and visible light images is fused using the fusion module VIFM according to the adaptive weights.

[0077] like Figure 2 As shown in the figure, this scheme retains YOLO's detection of three scales: large, medium and small, and can generate feature maps of three scales.

[0078] The neck network module is used to fuse features at different levels of the feature map. The fused feature map includes:

[0079] The neck network itself combines the design of PAN-FPN and forms a hierarchical feature fusion system with the SPPF and C2f modules to achieve the fusion of deep features and shallow features.

[0080] The BiFPN module uses a dynamic weighting mechanism to dynamically adjust the contribution of shallow and deep features.

[0081] More specifically, the method of using an efficient feature extraction module and a downsampling module to perform feature extraction includes:

[0082] The C2f_MS module uses multi-scale convolution operations to capture the detailed information of modal features.

[0083] The C2f_MS module replaces the Bottleneck module in C2f with the MSBlock module of the hierarchical feature fusion strategy, and achieves more comprehensive feature encoding through branch design and multi-scale convolution extraction.

[0084] The ADown module combines hybrid pooling and convolution operations to reduce the feature map resolution while maintaining spatial detail information.

[0085] The VIFM module improves the fusion effect of infrared and visible light modalities through adaptive weight distribution mechanism and channel reorganization operation.

[0086] Furthermore, the method of using multi-scale convolution operation to obtain detailed information of modal features is as follows:

[0087]

[0088] Among them, IB k×k represents the output of the inverse bottleneck layer with a kernel size of k×k, Y i-1 represents the output of the previous branch, X i is the i-th branch, Y i is the output of the i-th branch.

[0089] Furthermore, the method for processing modal features using an adaptive weight allocation mechanism and a channel reorganization operation includes:

[0090] Perform global average pooling on modal features to obtain global information;

[0091] Use the fully connected layer to process the global information and obtain the channel weight coefficient;

[0092] According to the weighting coefficient, obtain the weighted feature map;

[0093] Split the weighted feature map along the channel dimension to obtain the first channel and the second channel;

[0094] The first channel and the second channel are swapped and spliced ​​to obtain the processed modal features.

[0095] Specifically, the neck network uses the BiFPN module to efficiently fuse multi-scale features through a dynamic weighting strategy.

[0096] The backbone network adopts a dual-channel structure to process infrared and visible light modal features respectively, and realizes information complementarity through multiple modal fusions.

[0097] In the process of modal feature extraction, a lightweight module design is adopted to reduce the number of parameters and calculations and improve detection efficiency.

[0098] This embodiment also provides a UAV small target detection system based on dual-modal image fusion, including: an image acquisition subsystem and an image detection subsystem;

[0099] Image acquisition subsystem, used to collect images of the UAV under test;

[0100] The image detection subsystem is used to input the image of the drone to be tested into the target detection model to obtain the detection results, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module;

[0101] Backbone network module, used to obtain feature maps of different scales;

[0102] The neck network module is used to fuse features at different levels of the feature map to obtain the fused feature map;

[0103] The detection module is used to detect the fused feature map and obtain the detection results;

[0104] Output module, used to output detection results.

[0105] Specifically, this embodiment further provides a UAV small target detection system based on dual-modal image fusion, comprising: an image input module for receiving infrared and visible light modal images;

[0106] The feature extraction module adopts a dual-channel structure to extract features of infrared and visible light modes respectively;

[0107] Modal fusion module, proposes VIFM module to perform multiple modal feature fusion;

[0108] Feature optimization module, which uses C2f_MS and ADown modules to optimize feature extraction and downsampling;

[0109] Multi-scale feature fusion module, which uses the dynamic weighting mechanism of the BiFPN module to fuse features of different scales;

[0110] Detection module, outputs the result of target detection.

[0111] Furthermore, the backbone network module includes: convolution block, feature extraction unit, lightweight downsampling unit (downsampling module ADown), visible light infrared fusion unit (VIFM module) and splicing unit;

[0112] Convolutional block, used to extract local features of visible light images and infrared images, and perform normalization to obtain modal features of visible light images and infrared images;

[0113] A feature extraction unit is used to obtain detailed information of modal features using multi-scale convolution operations;

[0114] Lightweight downsampling unit, used to reduce the resolution of modal features;

[0115] A visible-light infrared fusion unit for processing modal features using an adaptive weight allocation mechanism and channel recombination operations;

[0116] The splicing unit is used to fuse and splice the processed modal features.

[0117] Furthermore, the method for processing modal features using an adaptive weight allocation mechanism and a channel reorganization operation includes:

[0118] Perform global average pooling on modal features to obtain global information;

[0119] Use the fully connected layer to process the global information and obtain the channel weight coefficient;

[0120] According to the weighting coefficient, obtain the weighted feature map;

[0121] Split the weighted feature map along the channel dimension to obtain the first channel and the second channel;

[0122] The first channel and the second channel are swapped and spliced ​​to obtain the processed modal features.

[0123] Specifically, the modal fusion module uses a weight normalization strategy to achieve adaptive adjustment of feature information, thereby enhancing the semantic expressiveness of the fused features. The detection module incorporates a lightweight design, supports efficient real-time detection, and demonstrates robust performance in a variety of complex scenarios. The final detection results include the target category, confidence level, and location coordinates.

[0124] The present embodiment will be described in detail below with reference to the accompanying drawings:

[0125] This embodiment proposes a dual-modal image fusion framework based on YOLOv8. First, the number of channels is expanded to form a dual-branch structure, and visible light and infrared images are input for processing respectively. In each branch, improved C2f_MS and ordinary CBS convolution are used to extract image features. In order to fully fuse the information of the two modalities, this paper proposes a VIFM fusion module to adaptively fuse the extracted features, and extract the fused results separately and then splice them into the neck network for deeper fusion. In the backbone network, multiple feature fusions are performed to integrate information from different modalities. In addition, the ADown module is used to replace part of the ordinary CBS module for downsampling, thereby improving the extraction of small target features. In the Neck network part, the weighted fusion mechanism of the bidirectional feature pyramid is used to fully fuse features of different scales, thereby improving the accuracy and robustness of target detection.

[0126] like Figure 1 As shown, this embodiment provides a method for detecting small targets of drones based on dual-modal image fusion, which specifically includes the following steps:

[0127] S1: Acquire visible light and infrared images of drone images;

[0128] Specifically, this example uses the DroneVehicle dataset for experiments. This dataset contains 28,439 pairs of infrared and visible light images collected by drones, including 17,990 pairs in the training set and 1,469 pairs in the validation set. The dataset divides images into five categories: "car," "truck," "bus," "van," and "freight car." The infrared image labels serve as the standard for both types of images, meaning that both infrared and visible light images share the same annotations. Specifically, there are 428,086 "car" images; 25,960 "truck" images; 16,590 "bus" images; 12,708 "van" images; and 17,173 "freight car" images.

[0129] S2: If Figure 2 As shown in the figure, the backbone network of YOLOv8 is designed as a dual-branch network, and the acquired images are input into the two branches respectively.

[0130] S3: Introduce efficient feature extraction modules and downsampling modules into the backbone network to obtain feature maps of different scales, and use the feature fusion module with adaptive weights to fuse the feature maps.

[0131] In this embodiment, the MSBlock module is introduced to replace the Bottenleck module in C2f, forming the C2f_MS module to enhance the ability to capture details and global features of infrared and visible light images.

[0132] The MSBlock module adopts a hierarchical feature fusion strategy, achieving more comprehensive feature encoding through branch design and multi-scale convolution extraction. Its core idea is to divide the input features into multiple branches and process each branch separately through the Inverse Bottleneck Layer (IB) with different convolution kernel sizes to fully encode information at different scales.

[0133] Assume that the input feature is X∈R H×W×C First, the number of channels is expanded to n*C by 1×1 convolution, and the expanded features are divided into n feature groups, which are represented as {X i}, where i∈{1,2,…,n}. For the first branch X1, the original feature information is retained directly, while the remaining branches are processed through the inverse bottleneck layer. In the inverse bottleneck layer, the features first undergo 1×1 convolution to achieve dimensionality reduction (maintaining resolution while adjusting the number of channels); then, depthwise separable convolution is used to efficiently extract spatial features; finally, 1×1 convolution is used to restore the number of channels. The features output by each branch can be expressed as follows:

[0134]

[0135] Among them, IB k×k represents the output of the inverse bottleneck layer with a kernel size of k×k, Y i-1 Represents the output of the previous branch. Through layer-by-layer residual connections, information transfer between different branches is ensured to avoid the loss of feature information.

[0136] Finally, the features of all branches are fused through a concatenation operation and integrated through a 1×1 convolution to obtain the final output.

[0137] In this embodiment, the ADown module is used to reduce the spatial dimension of the feature map while reducing the amount of computation.

[0138] Specifically, the input feature map is first downsampled using a global average pooling (AvgPool2d) operation to reduce spatial resolution while preserving global information. Subsequently, the feature map is divided into two chunks along the channel dimension: the first chunk further extracts local features through a standard convolutional layer (Conv3×3), while the second chunk captures regions of strong response through a maximum pooling operation (MaxPool2d) followed by a point-wise convolution (Conv1×1) to enhance nonlinear representation. Finally, the outputs of these two paths are concatenated along the channel dimension to form a downsampled result that fuses multi-scale features.

[0139] In this embodiment, to enhance the fusion of visible and infrared modal feature maps, a feature fusion module (VIFM) is employed. This module, through dynamic weighting and channel reorganization, preserves key details of each modality while enhancing semantic interaction between the modalities during feature extraction and fusion.

[0140] Specifically, if Figure 2 As shown in the figure, visible light and infrared features are extracted by the C2f_MSBlock and Conv downsampling modules, respectively, and then input into the VIFM module. VIFM first performs a dynamic weighted fusion of the two modal features, then implements feature interaction through channel reorganization, ultimately outputting a fused feature map. This processing method effectively integrates the complementary information of the two modalities and improves the model's semantic expression capabilities.

[0141] Specifically, if Figure 3 As shown in the figure, the workflow of the VIFM module consists of two main steps: feature weighting and channel reorganization. In the input feature map X, it is firstly processed by global average pooling (AP, Figure 3 The AvgPool2d) operation is used to obtain the global information of each channel. Assume that the input feature map X∈R B×C×H×W(Where B is the batch size, C is the number of channels, H and W are the height and width of the feature map respectively). After global average pooling, the feature map is compressed into a B×C×1×1 tensor, and Y represents the global information of each channel. This process can be expressed as follows:

[0142]

[0143] Among them, X b,c,h,w Represents the pixel value of the bth sample, cth channel, hth row, and wth column in the input feature map.

[0144] Next, the VIFM module processes the global information through a fully connected layer to obtain the weighted coefficients for each channel. This fully connected layer consists of two linear layers and an activation function (ReLU). It first reduces the dimensionality of the global information, then restores it to its original dimension, and finally generates the weighted coefficients through the Sigmoid activation function. The formula is:

[0145] W = σ(W2(ReLU(W1(Y))))

[0146] Among them, W1 and W2 represent the weight matrices of the fully connected layer, σ represents the Sigmoid function, and W is the calculated weighting coefficient, which represents the weight of each channel.

[0147] The weighting coefficient W is applied to the original feature map X to obtain a weighted feature map X1, in which the features of each channel are adjusted according to their importance. The purpose of the weighted operation is to retain the key detail information in each modality and provide a more expressive feature map for subsequent feature fusion. In order to further enhance the complementarity between modalities, the feature map X1 is split (Split) along the channel dimension to obtain two parts: the front C / / 2 and the back channel C / / 2. Subsequently, the channels in the second half are swapped with the channels in the first half (Swap). This helps to better fuse the information of different modalities, thereby enhancing the semantic expression ability of the two. Finally, the feature maps are spliced ​​through Concat, and the result is added to the input X to form a residual connection. This process can be expressed as:

[0148]

[0149] Among them, X o The final output feature map after weighting, segmentation, swapping, and concatenation. It retains the key spatial details of each modality while enhancing the semantic association between the two modalities through swapping and concatenation.

[0150] S4: The fusion results are extracted separately and then spliced ​​and input into the neck network. A bidirectional feature pyramid network is used to achieve efficient fusion of shallow graphic features and deep semantic features to obtain fused feature maps of different scales.

[0151] In this embodiment, the bidirectional feature pyramid network significantly enhances the network's ability to detect multi-scale targets through bidirectional information flow and a dynamic weighting mechanism, while providing a better solution for the efficient fusion of multimodal features.

[0152] Specifically, in order to reduce computational complexity, the present invention adopts a lightweight BiFPN_Concat2 module. BiFPN_Concat2 uses a dynamic weighting mechanism to fuse the input feature maps. Its core process is as follows: First, the module receives two layers of input feature maps and normalizes the weights using the following formula:

[0153]

[0154] Among them, ω i is a learning parameter, ε is the minimum value to prevent division by zero, and O is the weighted output.

[0155] By i Use ReLU activation function to ensure ω i ≥0, and to avoid numerical instability, ε is set to a minimum value of 0.0001, and the value of each normalized weight is between 0 and 1.

[0156] S5: Input the fused feature maps of different scales into the detector for detection and output the detection results.

[0157] This embodiment effectively addresses the issues of missed and false detections in small target detection caused by poor lighting conditions and small targets. The model detection results of this embodiment surpass those of early fusion methods such as MFEIF and PSFusion. Compared with late fusion methods such as BFDN_YOLOv8, the precision, recall, and mAP also improve by 2.8%, 3.9%, and 3.1%, respectively. This demonstrates that this invention is more adaptable to dual-modal image detection scenarios and meets the requirements of drone target detection.

[0158] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A small target detection method for UAV based on dual-modal image fusion, characterized in that: include: Obtain images of the drone to be tested; Input the image of the drone to be tested into a target detection model to obtain a detection result, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module; The backbone network module is used to obtain feature maps of different scales; The neck network module is used to fuse features at different levels on the feature map to obtain a fused feature map; The detection module is used to detect the fused feature map and obtain the detection result; The output module is used to output the detection result.

2. The method for detecting small targets on unmanned aerial vehicles based on dual-modal image fusion according to claim 1, characterized in that: The drone image to be tested includes: a visible light image and an infrared image.

3. The method for detecting small targets on unmanned aerial vehicles based on dual-modal image fusion according to claim 2, characterized in that: The backbone network module includes: a convolution block, a feature extraction unit, a lightweight downsampling unit, a visible light infrared fusion unit and a splicing unit; The convolution block is used to extract local features of the visible light image and the infrared image, and perform normalization to obtain modal features of the visible light image and the infrared image; The feature extraction unit is configured to obtain detailed information of the modal features by using a multi-scale convolution operation; The lightweight downsampling unit is used to reduce the resolution of the modal feature; The visible light infrared fusion unit is used to process the modal features using an adaptive weight allocation mechanism and a channel reorganization operation; The splicing unit is used to fuse and splice the processed modal features.

4. The method for detecting small targets on unmanned aerial vehicles based on dual-modal image fusion according to claim 3, characterized in that: The method of obtaining detailed information of the modal features using multi-scale convolution operation is as follows: Among them, IB k×k represents the output of the inverse bottleneck layer with a kernel size of k×k, Y i-1 represents the output of the previous branch, X i is the i-th branch, Y i is the output of the i-th branch.

5. The method for detecting small targets on unmanned aerial vehicles based on dual-modal image fusion according to claim 3, characterized in that: The method for processing the modality features using an adaptive weight allocation mechanism and a channel reorganization operation includes: Performing a global average pooling operation on the modal features to obtain global information; Processing the global information using a fully connected layer to obtain a weighted coefficient of the channel; Obtaining a weighted feature map according to the weighting coefficient; Split the weighted feature map along the channel dimension to obtain the first channel and the second channel; The first channel and the second channel are interchanged and spliced ​​to obtain processed modal features.

6. A UAV small target detection system based on dual-modal image fusion, characterized in that: include: Image acquisition subsystem and image detection subsystem; The image acquisition subsystem is used to acquire images of the drone to be tested; The image detection subsystem is used to input the image of the drone to be tested into a target detection model to obtain a detection result, wherein the target detection model includes: a backbone network module, a neck network module, a detection module and an output module; The backbone network module is used to obtain feature maps of different scales; The neck network module is used to fuse features at different levels on the feature map to obtain a fused feature map; The detection module is used to detect the fused feature map and obtain the detection result; The output module is used to output the detection result.

7. The UAV small target detection system based on dual-modal image fusion according to claim 6, characterized in that: The backbone network module includes: a convolution block, a feature extraction unit, a lightweight downsampling unit, a visible light infrared fusion unit and a splicing unit; The convolution block is used to extract local features of the visible light image and the infrared image, and perform normalization to obtain modal features of the visible light image and the infrared image; The feature extraction unit is configured to obtain detailed information of the modal features by using a multi-scale convolution operation; The lightweight downsampling unit is used to reduce the resolution of the modal feature; The visible light infrared fusion unit is used to process the modal features using an adaptive weight allocation mechanism and a channel reorganization operation; The splicing unit is used to fuse and splice the processed modal features.

8. The UAV small target detection system based on dual-modal image fusion according to claim 7, characterized in that: The method for processing the modality features using an adaptive weight allocation mechanism and a channel reorganization operation includes: Performing a global average pooling operation on the modal features to obtain global information; Processing the global information using a fully connected layer to obtain a weighted coefficient of the channel; Obtaining a weighted feature map according to the weighting coefficient; Split the weighted feature map along the channel dimension to obtain the first channel and the second channel; The first channel and the second channel are interchanged and spliced ​​to obtain processed modal features.

Images