An unmanned aerial vehicle small target detection method based on cross-modal multi-scale enhancement

Abstract

The application provides a kind of unmanned aerial vehicle small target detection method based on cross-modal multi-scale enhancement, by constructing infrared-visible light dual-mode detection network, the discrimination ability of fusion feature in complex background is improved, the precision of small target imaging detection is improved, and the missed detection rate is reduced;The scheme of the application is to set a cross-modal collaborative response enhancement module on multiple scales, and feed the enhanced fusion features back to the visible light branch and the infrared branch respectively, so that the network can maintain the modal independent representation ability while enhancing the cross-modal complementary information utilization ability;At the same time, the application retains a high-resolution small target detection branch in the detection head to enhance the positioning ability of the unmanned aerial vehicle small target.Compared with the existing infrared-visible light dual-mode detection method, the application is superior to the related indicators of the prior art in terms of comprehensive detection accuracy and target recall ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal target detection technology, and mainly relates to a method for detecting small targets on UAVs based on cross-modal multi-scale enhancement. Background Technology

[0002] Unmanned aerial vehicles (UAVs) possess advantages such as maneuverability, ease of deployment, and wide observation range, and have been widely applied in tasks such as low-altitude inspection, maritime surveillance, traffic monitoring, emergency rescue, and security reconnaissance. In these applications, UAVs typically need to perform real-time detection of targets on the ground, sea, and in the air. Due to the high imaging angle, long observation distance, and constantly changing flight attitude of UAV platforms, targets in images often appear small in scale, have weak appearance, significant morphological variations, and strong background interference. Since visible light images only have good target recognition capabilities under sufficient lighting and clear background conditions, infrared images have been widely adopted in industry and academia to address the degradation of visible light target features, demonstrating certain advantages in low light, nighttime, and some complex weather conditions. However, infrared images typically have low resolution and insufficient texture detail. Targets such as ships, aircraft, and UAVs appear as small spots or low-contrast areas when imaged at long distances, and are easily affected by heat source backgrounds, sea surface reflections, building thermal interference, and sensor noise. Single-modal images are difficult to stably complete target detection tasks under multiple scenes, categories, and scales. Therefore, how to achieve stable and accurate small target detection in UAV images is a key issue in low-altitude perception systems.

[0003] To improve the robustness of UAV target detection, infrared-visible dual-modal fusion detection has become an important technical approach. Existing infrared-visible fusion detection methods mainly include early stitching fusion, feature-level fusion, and decision-level fusion. Early stitching fusion methods typically stitch infrared and visible light images directly along the channel dimension and input them into the same detection network. This approach is structurally simple, but due to differences in the imaging mechanisms, grayscale distributions, noise characteristics, and target response patterns of the two modalities, direct stitching easily leads to modal information mixing, making it difficult for the network to learn visible light texture features and infrared thermal radiation features separately. Decision-level fusion methods typically detect the two modalities separately and then merge the detection results. This approach can preserve modal independence, but it is difficult to achieve fine-grained complementarity at the feature representation stage and is easily affected by single-modal detection errors. Feature-level fusion methods can interact between the features of the two modalities in the intermediate layers of the network and are currently a commonly used fusion method. However, existing methods still suffer from problems such as a single fusion location, a simple fusion method, and insufficient modeling of cross-modal correlations.

[0004] In UAV target detection scenarios, there are many target categories, large scale spans, and complex imaging postures. The feature responses of different target categories in different modalities vary significantly. Simple addition, simple stitching, or single-scale attention fusion is insufficient to fully explore the complementary relationship between infrared and visible light features. Furthermore, small targets are easily lost in deep feature maps after multiple network downsampling, leading to increased false negative rates and decreased localization accuracy. Conventional target detection networks often rely on mid-to-high-level semantic features for detection, which is beneficial for identifying large and medium-sized targets, but lacks sufficient spatial localization capability for small targets at long distances. For multi-category target detection tasks on UAV platforms, relying solely on deep, low-resolution features cannot simultaneously meet the requirements for small target localization and category discrimination. Therefore, existing technologies have at least the following shortcomings: First, infrared and visible light images exhibit cross-modal domain differences. Early stitching and fusion can easily lead to modal information mixing, making it difficult to fully retain the effective features of each modality. Existing feature fusion methods lack specialized modeling mechanisms for cross-modal feature complementarity and common target responses, making it difficult to highlight the effective information in both modalities that points to the real target, resulting in insufficient discriminative ability of fused features in complex backgrounds. Second, targets such as pedestrians, ships, vehicles and aircraft in long-distance imaging by UAVs often appear as small-sized, weak-textured or low-contrast targets. Existing detection networks are prone to losing spatial location information of small targets after multiple downsampling. Single-scale fusion is difficult to take into account shallow spatial details, mid-level structural features and deep semantic information, resulting in unstable detection and localization and missed detection problems. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, improve the discrimination ability of fused features in complex backgrounds, enhance the accuracy of small target imaging detection, and reduce the false negative rate, this invention provides a UAV small target detection method based on cross-modal multi-scale enhancement.

[0006] The purpose of this invention is to provide a method for small target detection in unmanned aerial vehicles (UAVs) based on cross-modal multi-scale enhancement, addressing the problems of insufficient cross-modal complementarity modeling, easy loss of small target features, and inadequate multi-scale fusion in existing bimodal target detection methods. This invention constructs an infrared-visible dual-branch feature extraction network, sets up cross-modal collaborative response enhancement modules at multiple scales, and feeds the enhanced fusion features back to the visible light branch and the infrared branch respectively, enabling the network to enhance its cross-modal complementary information utilization capability while maintaining its modal independent representation capability. Furthermore, this invention retains a high-resolution small target detection branch in the detection head to enhance the localization capability of small targets on UAVs.

[0007] The solution adopted by this invention to solve its technical problem includes the following steps:

[0008] Step S1: Construct a dataset; the dataset includes several infrared-visible image pairs; the several infrared-visible image pairs are visible light images and infrared images acquired in chronological order at the same observation location and in the same scene, and which have undergone preprocessing; each image in the infrared-visible image pair is labeled with the target category and the target bounding box;

[0009] Step S2: Construct an infrared-visible dual-modal detection network, and train the infrared-visible dual-modal detection network using a dataset. When the number of training rounds reaches the preset number of training rounds, or the decrease of the total loss function L for target detection is less than the preset threshold for several consecutive rounds, the optimal infrared-visible dual-modal detection network is obtained.

[0010] The infrared-visible dual-modal detection network includes a dual-branch backbone network, a multi-scale neck feature fusion module, and a detection head module. The dual-branch backbone network extracts features from both the visible light and infrared images, then fuses features at the same scale from both images to obtain second stitched features and third fused features at different scales. Fourth fusion feature and the fifth fusion feature Then combine the second splicing feature and the third fusion feature. Fourth fusion feature and the fifth fusion feature The input multi-scale neck feature fusion module uses a top-down and bottom-up bidirectional pyramid structure fusion process to transfer deep semantic information to high-resolution features and return shallow spatial detail information to the deep feature layer, obtaining the third, fourth, fifth, and sixth neck features. Finally, the third, fourth, fifth, and sixth neck features are input into the detection head module for classification and detection.

[0011] Step S3: Input the visible light image and infrared image to be detected into the optimal infrared-visible dual-modal detection network to obtain the detection results, which include the target bounding box, class probability and confidence level.

[0012] Furthermore, the preprocessing process is as follows:

[0013] The original visible light image and the original infrared image at the same observation location and in the same scene are acquired in chronological order. The acquired original visible light image and the original infrared image are paired and subjected to size unification and synchronous data enhancement processing to obtain infrared-visible light image pairs.

[0014] The synchronous data augmentation process includes one or more of random scaling, random flipping, random cropping, random translation, and Mosaic data augmentation.

[0015] Synchronous data augmentation applies the same geometric transformation to the target bounding boxes of both visible light and infrared images to ensure that the two modalities maintain their spatial correspondence after augmentation.

[0016] The target categories include pedestrians, ships, cars, buses, bicycles, airplanes, and drones; the bounding box Normalized coordinates are represented as:

[0017] ;

[0018] in, and Indicates the coordinates of the center point of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box.

[0019] Furthermore, the dual-branch backbone network includes a visible light feature extraction branch, an infrared feature extraction branch, a channel stitching module, a third cross-modal collaborative response enhancement module, a fourth cross-modal collaborative response enhancement module, and a fifth cross-modal collaborative response enhancement module; the input to the visible light feature extraction branch is a visible light image; the input to the infrared feature extraction branch is an infrared image.

[0020] The visible light feature extraction branch includes a first-stage module P1, a second-stage module P2, a third-stage module P3, a fourth-stage module P4, and a fifth-stage module P5 connected in series; the output of the visible light feature extraction branch is a high-resolution visible light feature. Medium resolution first feature Medium resolution second feature and semantic features The output of the second-stage module P2 features high-resolution visible light characteristics. The output of the third-stage module P3 is the first feature of visible light medium resolution. The output of the fourth-stage module P4 is the second feature of visible light medium resolution. The output of the fifth-stage module P5 is visible light semantic features. The first stage module P1, the second stage module P2, the third stage module P3, the fourth stage module P4, and the fifth stage module P5 all include sequentially connected convolutional downsampling layers Conv and C2f, which extract features of different resolutions step by step according to the network depth.

[0021] The structure of the infrared feature extraction branch is the same as that of the visible light feature extraction branch; the output of the infrared feature extraction branch is a high-resolution infrared feature. Infrared mid-resolution first feature Infrared mid-resolution second feature and infrared semantic features ;

[0022] The output of the second-stage module P2 of the infrared feature extraction branch is high-resolution infrared features. The output of the third-stage module P3 of the infrared feature extraction branch is the first infrared medium-resolution feature. The output of the fourth stage module P4 of the infrared feature extraction branch is the second infrared medium-resolution feature. The output of the fifth stage module P5 of the infrared feature extraction branch is infrared semantic features. ;

[0023] The input to the channel splicing module is visible light high-resolution features. and infrared high-resolution features The output of the channel splicing module is the second splicing feature;

[0024] The third, fourth, and fifth cross-modal cooperative response enhancement modules are all feature cooperative enhancement modules;

[0025] The input to the third cross-modal cooperative response enhancement module is the first feature of visible light medium resolution. and infrared mid-resolution first feature The output of the third cross-modal cooperative response enhancement module is the third fusion feature. ;

[0026] The input to the fourth cross-modal cooperative response enhancement module is the second feature of visible light medium resolution. and infrared mid-resolution second feature The output of the fourth cross-modal cooperative response enhancement module is the fourth fusion feature. ;

[0027] The input to the fifth cross-modal cooperative response enhancement module is visible light semantic features. and infrared semantic features The output of the fifth cross-modal cooperative response enhancement module is the fifth fusion feature. .

[0028] Furthermore, the feature collaboration enhancement module includes a channel stitching unit, a channel interaction unit, an attention unit, a dual-path projection output unit, a residual feedback enhancement module, and a feature stitching unit connected in sequence.

[0029] The input to the feature co-enhancement module includes visible light features. and infrared features ;

[0030] The channel splicing unit will incorporate visible light features. and infrared features Perform concatenation to obtain cross-modal joint features. :

[0031] ;

[0032] Cross-modal joint features The number of channels is C represents the visible light characteristics at scale s. and infrared features The number of channels for both is C. This indicates a channel-level concatenation operation;

[0033] The channel interaction unit comprises a first channel convolutional group, a second channel convolutional group, and a third channel convolutional group connected in sequence. Each channel convolutional group includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function. The input of the channel interaction unit is cross-modal joint features. The output of the channel interaction unit is a cross-modal intermediate feature. The first channel convolution, the second channel convolution, and the third channel convolution are all... Convolution; where , This represents the cross-modal channel interaction mapping function. Indicates intermediate features across modalities;

[0034] The attention unit comprises a global average pooling, a first attention convolution, a second attention convolution, and a sigmoid activation function connected in sequence.

[0035] The input to the attention unit is cross-modal intermediate features. The output of the attention unit is a cross-modal cooperative response feature. ;

[0036] ;

[0037] This represents element-wise multiplication. Indicates channel attention weights;

[0038] ;

[0039] ;

[0040] This indicates a global average pooling operation;

[0041] The dual-path projection output unit maps cross-modal cooperative response features. Mapped to visible light enhancement features respectively and infrared enhancement features ;

[0042] The input to the residual feedback enhancement module is the visible light enhancement feature. Infrared enhancement features Visible light characteristics and infrared features The outputs of the residual feedback enhancement module are respectively residual feedback enhanced visible light features. and residual feedback to enhance infrared features :

[0043] ;

[0044] ;

[0045] The input to the feature splicing unit is residual feedback enhanced visible light features. and residual feedback to enhance infrared features The feature splicing unit enhances visible light features through residual feedback. and residual feedback to enhance infrared features By splicing the data according to the channels, a fusion feature is obtained.

[0046] Furthermore, the multi-scale neck feature fusion module includes a top-down feature fusion module and a bottom-up feature fusion module;

[0047] The top-down feature fusion module includes a first upsampling fusion unit, a second upsampling fusion unit, and a third upsampling fusion unit;

[0048] The first upsampling fusion unit includes a first upsampling module and a first C2f module connected in sequence; the input of the first upsampling fusion unit includes a fifth fusion feature. and the fourth fusion feature The fifth fusion feature is used as the input to the first upsampling module. The output of the first upsampling module is concatenated with the fourth fusion feature channel and then input to the first C2f module. The output of the first C2f module is the first neck feature.

[0049] The second upsampling fusion unit includes a second upsampling module and a second C2f module connected in sequence; the input of the second upsampling fusion unit includes a first neck feature and a third fusion feature. The first neck feature is used as the input to the second upsampling module. The output of the second upsampling module is concatenated with the third fusion feature channel and then input to the second C2f module. The output of the second C2f module is the second neck feature.

[0050] The third upsampling fusion unit includes a third upsampling module and a third C2f module connected in sequence; the input of the third upsampling fusion unit includes a second neck feature and a second splicing feature; the second neck feature is used as the input of the third upsampling module, and the output of the third upsampling module is spliced ​​with the second splicing feature channel and then input into the third C2f module, and the output of the third C2f module is the third neck feature;

[0051] The bottom-up feature fusion module includes a first downsampling fusion unit, a second downsampling fusion unit, and a third downsampling fusion unit;

[0052] The first downsampling fusion unit includes a first downsampling module and a fourth C2f module connected in sequence; the input of the first downsampling fusion unit includes a third neck feature and a second neck feature; the third neck feature is used as the input of the first downsampling module, and the output of the first downsampling module is concatenated with the second neck feature channel and then input into the fourth C2f module, and the output of the fourth C2f module is the fourth neck feature;

[0053] The second downsampling fusion unit includes a second downsampling module and a fifth C2f module connected in sequence; the input of the second downsampling fusion unit includes a fourth neck feature and a first neck feature; the fourth neck feature is used as the input of the second downsampling module, and the output of the second downsampling module is concatenated with the first neck feature channel and then input into the fifth C2f module, and the output of the fifth C2f module is the fifth neck feature;

[0054] The third downsampling fusion unit includes a third downsampling module and a sixth C2f module connected in sequence; the input of the third downsampling fusion unit includes a fifth neck feature and a fifth fusion feature; the fifth neck feature is used as the input of the third downsampling module, and the output of the third downsampling module is concatenated with the fifth fusion feature channel and then input to the sixth C2f module, and the output of the sixth C2f module is the sixth neck feature.

[0055] Furthermore, the detection head module includes a small target detection head, a medium-small target detection head, a medium target detection head, a large target detection head, and a candidate result filtering module;

[0056] The input to the small target detection head is the third neck feature; the output of the small target detection head is the small target detection result.

[0057] The input to the small and medium-sized target detection head is the fourth neck feature; the output of the small and medium-sized target detection head is the small and medium-sized target detection result.

[0058] The input to the medium-sized target detection head is the fifth neck feature; the output of the medium-sized target detection head is the medium-sized target detection result.

[0059] The input to the large target detection head is the sixth neck feature; the output of the large target detection head is the large target detection result.

[0060] The large target detection results, medium target detection results, medium-small target detection results, and small target detection results all include target detection boxes, confidence scores, and class probabilities.

[0061] The candidate result screening module first calculates the product of confidence and class probability for large target detection results, medium target detection results, medium-small target detection results, and small target detection results, and retains the detection results with a product greater than a preset confidence threshold as candidate detection results. The candidate detection results are sorted from high to low according to the product of confidence and class probability, and non-maximum suppression is performed based on the intersection-union ratio between the predicted target detection boxes to remove target detection boxes whose overlap with the high-confidence predicted target detection boxes is greater than the preset intersection-union ratio threshold, thus obtaining the final detection result.

[0062] Furthermore, the total loss function L for object detection is:

[0063] ;

[0064] in, , , These represent the bounding box regression loss, the category classification loss, and the distribution focus loss, respectively. , and These are the weight coefficients for bounding box regression loss, category classification loss, and distribution focus loss, respectively.

[0065] The beneficial effects of this invention are:

[0066] This invention constructs an infrared-visible dual-branch backbone network, enabling feature extraction from visible and infrared images separately. This maintains the independent representation capabilities of the two modalities while avoiding the modal information mixing problem caused by early direct stitching. By setting up cross-modal collaborative response enhancement modules at multiple scales, complementary information and common target responses between infrared and visible features are modeled, and the cross-modal collaborative response information is injected into the corresponding modal branches through residual feedback, thereby enhancing the discriminative ability of fused features in complex backgrounds. By introducing a multi-scale neck feature fusion structure that includes a high-resolution small target detection branch, deep semantic information and shallow spatial detail information can be fully interacted, effectively alleviating the problem of small target features being easily lost during multiple downsampling processes, and improving the localization ability and detection stability of weak targets.

[0067] Compared with existing infrared-visible dual-modal detection methods, this invention achieves superior results in both overall detection accuracy and target recall capability. In specific embodiments, this invention achieves AP, AP50, AP75, and AR scores of 22.0%, 47.2%, 16.5%, and 36.6%, respectively, all higher than the comparative methods TarDAL, UA-CMDet, and CMA-Det. Specifically, compared to the best results among the comparative methods, this invention improves AP by 3.1 percentage points, AP50 by 6.3 percentage points, AP75 by 0.1 percentage points, and AR by 1.4 percentage points, indicating that this invention can improve the overall accuracy of small target detection by UAVs and reduce the risk of missed detections in complex scenarios. Attached Figure Description

[0068] Figure 1 This is a block diagram of a UAV small target detection method based on cross-modal multi-scale enhancement according to the present invention;

[0069] Figure 2 This is a schematic diagram of a UAV small target detection network structure based on cross-modal multi-scale enhancement according to the present invention;

[0070] Figure 3 This is a schematic diagram of a cross-modal collaborative response enhancement module structure according to the present invention. Detailed Implementation

[0071] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0072] This invention provides a cross-modal, multi-scale enhanced UAV small target detection method, the specific steps of which include:

[0073] Step 1: Acquire raw visible light images and raw infrared images of the same observation location and the same scene in chronological order. Pair the acquired raw visible light images and raw infrared images and perform size unification processing to obtain infrared-visible light image pairs. Each infrared-visible light image pair includes a visible light image and an infrared image. Construct a dataset based on all infrared-visible light image pairs. Each image in the infrared-visible light image pair is labeled with the target category and target bounding box.

[0074] Synchronous data enhancement is performed on infrared-visible image pairs; the synchronous data enhancement includes one or more of random scaling, random flipping, random cropping, random translation, and Mosaic data enhancement;

[0075] Synchronous data augmentation applies the same geometric transformation to both visible light and infrared images and bounding boxes to ensure that the two modalities retain their spatial correspondence after augmentation.

[0076] The target categories include pedestrians, ships, cars, buses, bicycles, airplanes, and drones; the bounding box Normalized coordinates are represented as:

[0077] ;

[0078] in, and Indicates the coordinates of the center point of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box;

[0079] Step 2: Construct an infrared-visible dual-modal detection network;

[0080] The infrared-visible dual-modal detection network includes a dual-branch backbone network, a multi-scale neck feature fusion module, and a detection head module;

[0081] The dual-branch backbone network includes a visible light feature extraction branch, an infrared feature extraction branch, a channel stitching module, a third cross-modal cooperative response enhancement module, a fourth cross-modal cooperative response enhancement module, and a fifth cross-modal cooperative response enhancement module. The input to the dual-branch backbone network is an infrared-visible light image pair; the input to the visible light feature extraction branch is a visible light image; the input to the infrared feature extraction branch is an infrared image; and the output of the dual-branch backbone network is a second stitching feature and a third fusion feature. Fourth fusion feature and the fifth fusion feature ;

[0082] The visible light feature extraction branch includes a first-stage module P1, a second-stage module P2, a third-stage module P3, a fourth-stage module P4, and a fifth-stage module P5 connected in series; the output of the visible light feature extraction branch is a high-resolution visible light feature. Medium resolution first feature Medium resolution second feature and semantic features The output of the second-stage module P2 features high-resolution visible light characteristics. The output of the third-stage module P3 is the first feature of visible light medium resolution. The output of the fourth-stage module P4 is the second feature of visible light medium resolution. The output of the fifth-stage module P5 is visible light semantic features. The first stage module P1, the second stage module P2, the third stage module P3, the fourth stage module P4, and the fifth stage module P5 all include sequentially connected convolutional downsampling layers Conv and C2f, which extract features of different resolutions step by step according to the network depth.

[0083] The structure of the infrared feature extraction branch is the same as that of the visible light feature extraction branch; the output of the infrared feature extraction branch is a high-resolution infrared feature. Infrared mid-resolution first feature Infrared mid-resolution second feature and infrared semantic features ;

[0084] The output of the second-stage module P2 of the infrared feature extraction branch is high-resolution infrared features. The output of the third-stage module P3 of the infrared feature extraction branch is the first infrared medium-resolution feature. The output of the fourth stage module P4 of the infrared feature extraction branch is the second infrared medium-resolution feature. The output of the fifth stage module P5 of the infrared feature extraction branch is infrared semantic features. ;

[0085] The input to the channel splicing module is visible light high-resolution features. and infrared high-resolution features The output of the channel splicing module is the second splicing feature;

[0086] The third, fourth, and fifth cross-modal cooperative response enhancement modules are all feature cooperative enhancement modules;

[0087] The input to the third cross-modal cooperative response enhancement module is the first feature of visible light medium resolution. and infrared mid-resolution first feature The output of the third cross-modal cooperative response enhancement module is the third fusion feature. ;

[0088] The input to the fourth cross-modal cooperative response enhancement module is the second feature of visible light medium resolution. and infrared mid-resolution second feature The output of the fourth cross-modal cooperative response enhancement module is the fourth fusion feature. ;

[0089] The input to the fifth cross-modal cooperative response enhancement module is visible light semantic features. and infrared semantic features The output of the fifth cross-modal cooperative response enhancement module is the fifth fusion feature. ;

[0090] The feature collaboration enhancement module includes a channel stitching unit, a channel interaction unit, an attention unit, a dual-path projection output unit, a residual feedback enhancement module, and a feature stitching unit connected in sequence.

[0091] The input to the feature co-enhancement module includes visible light features. and infrared features ;

[0092] The channel splicing unit will incorporate visible light features. and infrared features Perform concatenation to obtain cross-modal joint features. :

[0093] ;

[0094] Cross-modal joint features The number of channels is C represents the visible light characteristics at scale s. and infrared features The number of channels for both is C. This indicates a channel-level concatenation operation;

[0095] The channel interaction unit comprises a first channel convolutional group, a second channel convolutional group, and a third channel convolutional group connected in sequence. Each channel convolutional group includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function. The input of the channel interaction unit is cross-modal joint features. The output of the channel interaction unit is a cross-modal intermediate feature. The first channel convolution, the second channel convolution, and the third channel convolution are all... Convolution; where , This represents the cross-modal channel interaction mapping function. Indicates intermediate features across modalities;

[0096] The attention unit comprises a global average pooling, a first attention convolution, a second attention convolution, and a sigmoid activation function connected in sequence.

[0097] The input to the attention unit is cross-modal intermediate features. The output of the attention unit is a cross-modal cooperative response feature. ;

[0098] ;

[0099] This represents element-wise multiplication. Indicates channel attention weights;

[0100] ;

[0101] ;

[0102] This indicates a global average pooling operation;

[0103] The dual-path projection output unit maps cross-modal cooperative response features. Mapped to visible light enhancement features respectively and infrared enhancement features ;

[0104] The input to the residual feedback enhancement module is the visible light enhancement feature. Infrared enhancement features Visible light characteristics and infrared features The outputs of the residual feedback enhancement module are respectively residual feedback enhanced visible light features. and residual feedback to enhance infrared features :

[0105] ;

[0106] ;

[0107] The input to the feature splicing unit is residual feedback enhanced visible light features. and residual feedback to enhance infrared features The feature splicing unit enhances visible light features through residual feedback. and residual feedback to enhance infrared features The data is stitched together according to the channels to obtain fusion features;

[0108] The multi-scale neck feature fusion module includes a top-down feature fusion module and a bottom-up feature fusion module;

[0109] The top-down feature fusion module includes a first upsampling fusion unit, a second upsampling fusion unit, and a third upsampling fusion unit;

[0110] The first upsampling fusion unit includes a first upsampling module and a first C2f module connected in sequence; the input of the first upsampling fusion unit includes a fifth fusion feature. and the fourth fusion feature The fifth fusion feature is used as the input to the first upsampling module. The output of the first upsampling module is concatenated with the fourth fusion feature channel and then input to the first C2f module. The output of the first C2f module is the first neck feature.

[0111] The second upsampling fusion unit includes a second upsampling module and a second C2f module connected in sequence; the input of the second upsampling fusion unit includes a first neck feature and a third fusion feature. The first neck feature is used as the input to the second upsampling module. The output of the second upsampling module is concatenated with the third fusion feature channel and then input to the second C2f module. The output of the second C2f module is the second neck feature.

[0112] The third upsampling fusion unit includes a third upsampling module and a third C2f module connected in sequence; the input of the third upsampling fusion unit includes a second neck feature and a second splicing feature; the second neck feature is used as the input of the third upsampling module, and the output of the third upsampling module is spliced ​​with the second splicing feature channel and then input into the third C2f module, and the output of the third C2f module is the third neck feature;

[0113] The bottom-up feature fusion module includes a first downsampling fusion unit, a second downsampling fusion unit, and a third downsampling fusion unit;

[0114] The first downsampling fusion unit includes a first downsampling module and a fourth C2f module connected in sequence; the input of the first downsampling fusion unit includes a third neck feature and a second neck feature; the third neck feature is used as the input of the first downsampling module, and the output of the first downsampling module is concatenated with the second neck feature channel and then input into the fourth C2f module, and the output of the fourth C2f module is the fourth neck feature;

[0115] The second downsampling fusion unit includes a second downsampling module and a fifth C2f module connected in sequence; the input of the second downsampling fusion unit includes a fourth neck feature and a first neck feature; the fourth neck feature is used as the input of the second downsampling module, and the output of the second downsampling module is concatenated with the first neck feature channel and then input into the fifth C2f module, and the output of the fifth C2f module is the fifth neck feature;

[0116] The third downsampling fusion unit includes a third downsampling module and a sixth C2f module connected in sequence; the input of the third downsampling fusion unit includes a fifth neck feature and a fifth fusion feature; the fifth neck feature is used as the input of the third downsampling module, and the output of the third downsampling module is concatenated with the fifth fusion feature channel and then input into the sixth C2f module, and the output of the sixth C2f module is the sixth neck feature;

[0117] The detection head module includes a small target detection head, a medium-small target detection head, a medium target detection head, a large target detection head, and a candidate result filtering module;

[0118] The input to the small target detection head is the third neck feature; the output of the small target detection head is the small target detection result.

[0119] The input to the small and medium-sized target detection head is the fourth neck feature; the output of the small and medium-sized target detection head is the small and medium-sized target detection result.

[0120] The input to the medium-sized target detection head is the fifth neck feature; the output of the medium-sized target detection head is the medium-sized target detection result.

[0121] The input to the large target detection head is the sixth neck feature; the output of the large target detection head is the large target detection result.

[0122] The large target detection results, medium target detection results, medium-small target detection results, and small target detection results all include target detection boxes, confidence scores, and class probabilities.

[0123] The candidate result screening module first calculates the product of confidence and class probability for large target detection results, medium target detection results, medium-small target detection results, and small target detection results, and retains the detection results with a product greater than a preset confidence threshold as candidate detection results; the candidate detection results are sorted from high to low according to the product of confidence and class probability, and non-maximum suppression is performed based on the intersection-union ratio between the predicted target detection boxes to remove target detection boxes whose overlap with high-confidence predicted target detection boxes is greater than the preset intersection-union ratio threshold, thus obtaining the final detection result;

[0124] This invention uses a total loss function L for target detection to optimize the network; the total loss function L is:

[0125] ;

[0126] in, , , These represent the bounding box regression loss, the category classification loss, and the distribution focus loss, respectively. , and These represent the weight coefficients of the corresponding loss terms. The bounding box regression loss constrains the positional differences between the predicted and ground truth boxes; the class classification loss constrains the differences between the predicted and ground truth classes; and the distribution focus loss improves the granularity of the bounding box regression.

[0127] Training completion criteria include at least one of the following: reaching a preset number of training epochs; the validation set loss decreasing by less than a preset threshold over several consecutive epochs; the average accuracy of the validation set increasing by less than a preset threshold over several consecutive epochs; and the average accuracy of the validation set meeting a preset requirement. Once the training completion criteria are met, the network weights with the best performance on the validation set are saved as the final detection model.

[0128] As an embodiment of the present invention, the specific steps are as follows:

[0129] Step 1: In this embodiment, the publicly available dataset RGBT-Tiny is used as the infrared-visible dual-modal target detection dataset, with a resolution of [resolution missing]. The image dataset comprises 46,701 pairs of pixels, of which 34,111 pairs were used for training and 12,590 pairs were used for both training and testing. In one specific embodiment, the two modalities of the images are uniformly adjusted to... Pixels. Subsequently, the visible light image and the infrared image are used as two inputs to the dual-modal detection network, respectively, and fed into the visible light feature extraction branch and the infrared feature extraction branch.

[0130] During the training phase, simultaneous data augmentation can be performed on visible light and infrared images. Simultaneous data augmentation refers to performing the same geometric transformation on the same set of visible light and infrared images and their bounding boxes to ensure that the spatial correspondence between the two modalities is maintained after augmentation. The simultaneous data augmentation can include one or more of random scaling, random flipping, random cropping, random translation, and Mosaic data augmentation. The target annotation file includes target categories and target bounding boxes. Target categories include seven types: pedestrians, ships, cars, buses, bicycles, airplanes, and drones. The bounding boxes can be represented as normalized coordinates.

[0131]

[0132] in, and Indicates the coordinates of the center point of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box.

[0133] Step 2: The infrared-visible dual-modal detection network in this embodiment includes a dual-branch backbone network, a cross-modal collaborative response enhancement module, a residual feedback enhancement module, a multi-scale neck feature fusion module, and a detection head module. The visible light image is input to the visible light feature extraction branch, and the infrared image is input to the infrared feature extraction branch. The two branches extract multi-scale features corresponding to their respective modalities. The same-scale features output from the two branches are sequentially processed by the cross-modal collaborative response enhancement module and the residual feedback enhancement module to form enhanced features. The enhanced multi-scale features are input to the multi-scale neck feature fusion structure for cross-scale fusion, and finally, the prediction module outputs the target bounding box, target category, and confidence score.

[0134] Step 2.1: In this embodiment, both the visible light feature extraction branch and the infrared feature extraction branch of the dual-branch backbone network adopt the CSPDarknet backbone network, including a convolutional downsampling layer, a C2f feature extraction module, and a fast spatial pyramid pooling module. The two branches sequentially perform convolutional downsampling and feature extraction on the visible light image to obtain P2, P3, P4, and P5 multi-scale visible light and infrared features. The P2 feature layer has high spatial resolution and is used to preserve the edges, contours, and positional details of small targets; the P3 and P4 feature layers are used to extract the mid-level structural information of the target; and the P5 feature layer is used to extract deep semantic discrimination information. In one specific implementation, the visible light branch and the infrared branch generate the following multi-scale features:

[0135] , , ,

[0136] , , ,

[0137] in, , , , These represent the characteristics of the visible light branch output at the P2, P3, P4, and P5 scales, respectively. , , , These represent the characteristics of the infrared branch output at the P2, P3, P4, and P5 scales, respectively.

[0138] The two branches maintain a structural correspondence, but their parameters are independent. This allows the visible light branch to focus on learning texture, edge, color, and morphological information, while the infrared branch focuses on learning thermal radiation response and low-light saliency information.

[0139] Step 2.2: In this embodiment, cross-modal cooperative response enhancement modules are set up at three scales: P3, P4, and P5. Each cross-modal cooperative response enhancement module receives visible light and infrared features at the same scale as input. For each scale... The input to the cross-modal cooperative response enhancement module is:

[0140]

[0141]

[0142] in, Indicates the feature scale. Indicates the number of channels. Representing scale The feature map height below, Representing scale The width of the feature map below.

[0143] The cross-modal collaborative response enhancement module includes a channel splicing unit, a cross-modal channel interaction unit, a channel attention unit, and a dual-path projection output unit.

[0144] First, the channel stitching unit stitches visible light features and infrared features of the same scale along the channel dimension to obtain cross-modal joint features:

[0145]

[0146] in, This represents a cross-modal joint feature, with the number of channels being... , This indicates a channel dimension splicing operation.

[0147] Then, the cross-modal channel interaction unit performs channel interaction and channel compression on the cross-modal joint features. In one specific implementation, the cross-modal channel interaction unit includes three sets of... Convolution, batch normalization, and ReLU activation function. First group. Convolution reduces the number of channels from Compress to The next two groups Convolution preserves the number of channels. We obtain cross-modal intermediate features:

[0148]

[0149] in, This represents the cross-modal channel interaction mapping function. This represents an intermediate feature across modalities.

[0150] Subsequently, the channel attention unit assigns channel weights to the cross-modal intermediate features. First, it... Perform global average pooling to obtain the channel description vector:

[0151]

[0152] in, This indicates a global average pooling operation.

[0153] Then, through two layers Convolution and Sigmoid activation functions generate channel attention weights:

[0154]

[0155] in, and express Convolution mapping, Represents the ReLU activation function. Sigmoid activation function This represents the channel attention weight.

[0156] Then, the channel attention weights are applied to the cross-modal intermediate features to obtain the cross-modal collaborative response features:

[0157]

[0158] in, This represents element-wise multiplication. This represents the cross-modal cooperative response characteristics after channel attention recalibration.

[0159] Finally, the dual-projection output unit maps the cross-modal cooperative response features to visible light enhancement features and infrared enhancement features, respectively. Specifically, the visible light projection layer generates enhancement features oriented towards the visible light branch:

[0160]

[0161] The infrared projection layer generates enhanced features oriented towards the infrared branch:

[0162]

[0163] in, Represents visible light projection mapping, Indicates infrared projection mapping, Indicates visible light enhancement characteristics, This indicates infrared enhancement features.

[0164] In this embodiment, the cross-modal cooperative response enhancement modules are set at scales P3, P4, and P5, respectively. The P3 scale is used to enhance the local response and weak texture information of small targets; the P4 scale is used to enhance the target structural information and cross-modal consistent response; and the P5 scale is used to enhance deep semantic discrimination information.

[0165] Step 2.3: The residual feedback enhancement module is used to feed back the dual-path enhancement features output by the cross-modal collaborative response enhancement module to the corresponding modal branches. This module uses a residual summation connection method to inject cross-modal collaborative response information while preserving the original modal features.

[0166] For scale The residual feedback enhancement process of the visible light branch is as follows:

[0167]

[0168] The residual feedback enhancement process of the infrared branch is as follows:

[0169]

[0170] in, This indicates the visible light characteristics after residual feedback enhancement. This represents the infrared characteristics after residual feedback enhancement.

[0171] Through residual feedback enhancement, the network can inject the cooperative responses from both modalities that point to the target region into the visible light branch and the infrared branch respectively without destroying the original modal feature representation, thereby improving the ability to utilize cross-modal complementary information.

[0172] In this embodiment, residual feedback enhancement modules are set at scales P3, P4 and P5 to provide feedback enhancement for visible light features and infrared features at the corresponding scales, respectively.

[0173] Step 2.4: After completing the cross-modal cooperative response enhancement and residual feedback enhancement, the same-scale features of the visible light branch and the infrared branch are fused to obtain multi-scale dual-modal fused features.

[0174] For the P2 scale, since this scale has high spatial resolution, this embodiment directly stitches together the visible light P2 features and the infrared P2 features:

[0175]

[0176] For P3, P4, and P5 scales, channel stitching is performed using visible light and infrared features enhanced by residual feedback:

[0177]

[0178]

[0179]

[0180] The multi-scale neck feature fusion structure receives the aforementioned P2, P3, P4, and P5 fused features and performs cross-scale information interaction through a bidirectional fusion process of top-down and bottom-up. In the top-down path, the P5 fused feature is first upsampled and concatenated with the P4 fused feature, then passed through a convolutional enhancement module to obtain the enhanced P4 detection feature. Subsequently, the enhanced P4 detection feature is upsampled and concatenated with the P3 fused feature, then passed through a convolutional enhancement module to obtain the enhanced P3 detection feature. The enhanced P3 detection feature is then upsampled again and concatenated with the P2 fused feature, then passed through a convolutional enhancement module to obtain the enhanced P2 detection feature. In the bottom-up path, the enhanced P2 detection feature is downsampled and concatenated with the enhanced P3 detection feature, then passed through a convolutional enhancement module to obtain the returned P3 detection feature. Subsequently, the returned P3 detection feature is downsampled and concatenated with the enhanced P4 detection feature, then passed through a convolutional enhancement module to obtain the returned P4 detection feature.

[0181] Through the multi-scale neck feature fusion structure described above, the network can transmit deep semantic information to the high-resolution feature layer and transmit shallow spatial detail information back to the deep feature layer, thereby simultaneously enhancing the ability to locate small targets and the ability to distinguish semantics of multi-category targets.

[0182] Step 2.4: The prediction module receives the multi-scale detection features output by the multi-scale neck feature fusion structure and generates the target detection results. The prediction results include the target bounding box, target category, and confidence score.

[0183] In this embodiment, the prediction module includes at least one high-resolution small target detection branch. This high-resolution small target detection branch receives detection features obtained by fusing with the P2 layer, which is used to enhance the spatial localization capability of small targets. Other prediction branches receive deeper detection features, which are used to enhance the semantic recognition capability of targets at different scales.

[0184] The prediction module outputs bounding box location, class probability, and confidence score for each candidate target. The bounding box location represents the target's spatial coordinates in the image, the class probability is used to determine the target's class, and the confidence score indicates the probability that the target exists within the predicted bounding box.

[0185] The prediction module ultimately outputs the detection set:

[0186]

[0187] in, Represents the set of detection outputs. Indicates the first The bounding box of each target. Indicates the first Confidence level of each target Indicates the first The categories of targets, This indicates the number of targets detected.

[0188] Step 2.5: During the training phase, the preprocessed infrared-visible image pairs are input into the bimodal detection network, with target category and target bounding box used as supervision information. Training employs an end-to-end optimization using the target detection loss function. The total loss function includes bounding box regression loss, category classification loss, and distribution focus loss:

[0189]

[0190] in, Represents the total loss function. , , These represent the bounding box regression loss, the category classification loss, and the distribution focus loss, respectively. , and These represent the weight coefficients of the corresponding loss terms. The bounding box regression loss constrains the positional differences between the predicted and ground truth boxes. The class classification loss constrains the differences between the predicted and ground truth classes. The distribution focus loss improves the granularity of the bounding box regression.

[0191] In one specific implementation, the training batch size is set to 36, the number of training epochs is set to 100, the optimizer is set to AdamW, and the initial learning rate is set to 0.0005. These parameters can be adjusted according to actual hardware conditions and dataset size. Training ends after 100 epochs, and the network weights with the best performance on the validation set are saved as the final detection model.

[0192] Step 2.6: After model training is complete, validate the model's performance using a validation set or test set. During validation, the visible light image and infrared image to be detected are input into the model, and the model outputs the target bounding box, category, and confidence score. This invention uses COCO-style object detection evaluation metrics to calculate model performance, including the following evaluation metrics:

[0193] Intersecton over Union (IoU) is a commonly used metric in calculating the performance of object detection models. It measures the degree of overlap between the model's detection results and the actual annotations, and is also known as the overlap ratio. Its calculation formula is as follows:

[0194]

[0195] in, and This represents the predicted bounding box and the ground truth bounding box.

[0196] For a given threshold If a predicted bounding box is close to the true bounding box of the same class... Greater than or equal to If a predicted bounding box is not matched by any other predicted bounding box, it is designated as a TruePositive (TP). If a predicted bounding box fails to match any true bounding box of the same class, it is designated as a False Positive (FP). If a true target is not matched by any predicted bounding box, it is designated as a False Negative (FN). Given a confidence threshold, precision and recall are calculated as follows:

[0197]

[0198]

[0199] In the AP calculation process, the predicted boxes output by the model are first sorted from high to low confidence, and then the confidence thresholds are changed sequentially to obtain a set of precision-recall curves. For a certain category... and a certain threshold The average precision of this category at this threshold is denoted as:

[0200]

[0201] in, Indicate category and in threshold Below, when the recall rate is The corresponding accuracy.

[0202] In practical calculations, discrete interpolation is used to approximate the above integral. COCO style evaluation typically falls within the recall range. Take 101 sampling points, denoted as:

[0203]

[0204] Then a certain category and in threshold The average precision can be expressed as:

[0205]

[0206] in, This means that the recall rate is not less than The maximum accuracy achieved within the specified range is calculated as follows:

[0207]

[0208] Indicators represent multiple Threshold and average accuracy across multiple target categories. COCO style. Adopted The threshold set is:

[0209]

[0210] Therefore, the whole It can be represented as:

[0211]

[0212] in, express Threshold set, express Number of thresholds Represents the set of target categories. Indicates the number of categories.

[0213] express The average accuracy when the threshold is fixed at 0.50 is calculated as follows:

[0214]

[0215] express The average accuracy when the threshold is fixed at 0.75 is calculated as follows:

[0216]

[0217] The average recall rate is calculated using multiple... The ability of a threshold-based statistical model to recall true targets, and to handle different... The threshold and different categories are averaged. In this embodiment, The average recall rate is calculated when a maximum of 100 predicted boxes are retained per image. For each category... and in threshold The recall rate is denoted as:

[0218]

[0219] in, Indicate category and in threshold The number of actual targets that were correctly detected. Indicate category The total number of actual targets.

[0220] Therefore, AR can be represented as:

[0221]

[0222] in, , This represents the set of target categories.

[0223] In this embodiment, the model output includes the target category, predicted bounding box, and confidence score. After converting the prediction results and ground truth labels into COCO format, the calculation is performed according to the above method. , , and Indicators. All indicators are expressed as percentages.

[0224] In one specific embodiment, the present invention is compared with TarDAL, a representative algorithm for early-stage visible-infrared dual-modal fusion; UA-CMDet, a representative algorithm for mid-stage visible-infrared dual-modal fusion; and CMA-Det, a visible-infrared dual-modal UAV target detection algorithm. Table 1 presents the experimental results. As can be seen from the table, the present invention achieves [the following results]: , , and The results achieved in all indicators were superior, at 22.0%, 47.2%, 16.5%, and 36.6%, respectively, all higher than the other three methods, demonstrating the advantages of this invention in comprehensive detection accuracy and strict bounding box positioning detection.

[0225] Table 1. Experimental results comparing the present invention with other methods.

[0226]

Claims

1. A method for detecting small targets on unmanned aerial vehicles (UAVs) based on cross-modal multi-scale enhancement, characterized in that, Includes the following steps: Step S1: Construct a dataset; the dataset includes several infrared-visible image pairs; the several infrared-visible image pairs are visible light images and infrared images acquired in chronological order at the same observation location and in the same scene, and which have undergone preprocessing; each image in the infrared-visible image pair is labeled with the target category and the target bounding box; Step S2: Construct an infrared-visible dual-modal detection network, and train the infrared-visible dual-modal detection network using a dataset. When the number of training rounds reaches the preset number of training rounds, or the decrease of the total loss function L for target detection is less than the preset threshold for several consecutive rounds, the optimal infrared-visible dual-modal detection network is obtained. The infrared-visible dual-modal detection network includes a dual-branch backbone network, a multi-scale neck feature fusion module, and a detection head module. The dual-branch backbone network extracts features from both the visible light and infrared images, then fuses features at the same scale from both images to obtain second stitched features and third fused features at different scales. Fourth fusion feature and the fifth fusion feature Then combine the second splicing feature and the third fusion feature. Fourth fusion feature and the fifth fusion feature The input multi-scale neck feature fusion module uses a top-down and bottom-up bidirectional pyramid structure fusion process to transfer deep semantic information to high-resolution features and return shallow spatial detail information to the deep feature layer, obtaining the third, fourth, fifth, and sixth neck features. Finally, the third, fourth, fifth, and sixth neck features are input into the detection head module for classification and detection. Step S3: Input the visible light image and infrared image to be detected into the optimal infrared-visible dual-modal detection network to obtain the detection results, which include the target bounding box, class probability and confidence level.

2. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 1, characterized in that, Furthermore, the dual-branch backbone network includes a visible light feature extraction branch, an infrared feature extraction branch, a channel splicing module, a third cross-modal collaborative response enhancement module, a fourth cross-modal collaborative response enhancement module, and a fifth cross-modal collaborative response enhancement module; The input to the visible light feature extraction branch is a visible light image; The input to the infrared feature extraction branch is an infrared image; The visible light feature extraction branch includes a first-stage module P1, a second-stage module P2, a third-stage module P3, a fourth-stage module P4, and a fifth-stage module P5 connected in series; the output of the visible light feature extraction branch is a high-resolution visible light feature. Medium resolution first feature Medium resolution second feature and semantic features The output of the second-stage module P2 features high-resolution visible light characteristics. ; The output of the third-stage module P3 is the first feature of visible light medium resolution. The output of the fourth-stage module P4 is the second feature of visible light medium resolution. ; The output of the fifth-stage module P5 is visible light semantic features. The first stage module P1, the second stage module P2, the third stage module P3, the fourth stage module P4, and the fifth stage module P5 all include sequentially connected convolutional downsampling layers Conv and C2f, which extract features of different resolutions step by step according to the network depth. The structure of the infrared feature extraction branch is the same as that of the visible light feature extraction branch; the output of the infrared feature extraction branch is a high-resolution infrared feature. Infrared mid-resolution first feature Infrared mid-resolution second feature and infrared semantic features ; The output of the second-stage module P2 of the infrared feature extraction branch is high-resolution infrared features. The output of the third-stage module P3 of the infrared feature extraction branch is the first infrared medium-resolution feature. The output of the fourth stage module P4 of the infrared feature extraction branch is the second infrared medium-resolution feature. The output of the fifth stage module P5 of the infrared feature extraction branch is infrared semantic features. ; The input to the channel splicing module is visible light high-resolution features. and infrared high-resolution features The output of the channel splicing module is the second splicing feature; The third, fourth, and fifth cross-modal cooperative response enhancement modules are all feature cooperative enhancement modules; The input to the third cross-modal cooperative response enhancement module is the first feature of visible light medium resolution. and infrared mid-resolution first feature The output of the third cross-modal cooperative response enhancement module is the third fusion feature. ; The input to the fourth cross-modal cooperative response enhancement module is the second feature of visible light medium resolution. and infrared mid-resolution second feature The output of the fourth cross-modal cooperative response enhancement module is the fourth fusion feature. ; The input to the fifth cross-modal cooperative response enhancement module is visible light semantic features. and infrared semantic features The output of the fifth cross-modal cooperative response enhancement module is the fifth fusion feature. .

3. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 2, characterized in that, The feature collaboration enhancement module includes a channel stitching unit, a channel interaction unit, an attention unit, a dual-path projection output unit, a residual feedback enhancement module, and a feature stitching unit connected in sequence. The input to the feature co-enhancement module includes visible light features. and infrared features ; The channel splicing unit will incorporate visible light features. and infrared features By concatenating the features, we can obtain cross-modal joint features. : ； Cross-modal joint features The number of channels is C represents the visible light characteristics at scale s. and infrared features The number of channels for both is C. This indicates a channel-level concatenation operation; The channel interaction unit comprises a first channel convolutional group, a second channel convolutional group, and a third channel convolutional group connected in sequence. Each channel convolutional group includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function. The input of the channel interaction unit is cross-modal joint features. The output of the channel interaction unit is a cross-modal intermediate feature. The first channel convolution, the second channel convolution, and the third channel convolution are all... Convolution; where , This represents the cross-modal channel interaction mapping function. Indicates intermediate features across modalities; The attention unit comprises a global average pooling, a first attention convolution, a second attention convolution, and a sigmoid activation function connected in sequence. The input to the attention unit is cross-modal intermediate features. The output of the attention unit is a cross-modal cooperative response feature. ; ； This represents element-wise multiplication. Indicates channel attention weights; ； ； This indicates a global average pooling operation; The dual-path projection output unit maps cross-modal cooperative response features. Mapped to visible light enhancement features respectively and infrared enhancement features ; The input to the residual feedback enhancement module is the visible light enhancement feature. Infrared enhancement features Visible light characteristics and infrared features The outputs of the residual feedback enhancement module are respectively residual feedback enhanced visible light features. and residual feedback to enhance infrared features : ； ； The input to the feature splicing unit is residual feedback enhanced visible light features. and residual feedback to enhance infrared features ;feature The splicing unit enhances visible light characteristics through residual feedback. and residual feedback to enhance infrared features By splicing the data according to the channels, a fusion feature is obtained.

4. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 1, characterized in that, The multi-scale neck feature fusion module includes a top-down feature fusion module and a bottom-up feature fusion module; The top-down feature fusion module includes a first upsampling fusion unit, a second upsampling fusion unit, and a third upsampling fusion unit; The first upsampling fusion unit includes a first upsampling module and a first C2f module connected in sequence; The input to the first upsampling fusion unit includes the fifth fusion feature. and the fourth fusion feature The fifth fusion feature is used as the input to the first upsampling module. The output of the first upsampling module is concatenated with the fourth fusion feature channel and then input to the first C2f module. The output of the first C2f module is the first neck feature. The second upsampling fusion unit includes a second upsampling module and a second C2f module connected in sequence; the input of the second upsampling fusion unit includes a first neck feature and a third fusion feature. The first neck feature is used as the input to the second upsampling module. The output of the second upsampling module is concatenated with the third fusion feature channel and then input to the second C2f module. The output of the second C2f module is the second neck feature. The third upsampling fusion unit includes a third upsampling module and a third C2f module connected in sequence; the input of the third upsampling fusion unit includes a second neck feature and a second splicing feature; the second neck feature is used as the input of the third upsampling module, and the output of the third upsampling module is spliced ​​with the second splicing feature channel and then input into the third C2f module, and the output of the third C2f module is the third neck feature; The bottom-up feature fusion module includes a first downsampling fusion unit, a second downsampling fusion unit, and a third downsampling fusion unit; The first downsampling fusion unit includes a first downsampling module and a fourth C2f module connected in sequence; the input of the first downsampling fusion unit includes a third neck feature and a second neck feature; The third neck feature is used as the input of the first downsampling module. The output of the first downsampling module is concatenated with the second neck feature channel and then input into the fourth C2f module. The output of the fourth C2f module is the fourth neck feature. The second downsampling fusion unit includes a second downsampling module and a fifth C2f module connected in sequence; the input of the second downsampling fusion unit includes a fourth neck feature and a first neck feature; The fourth neck feature is used as the input of the second downsampling module. The output of the second downsampling module is concatenated with the first neck feature channel and then input into the fifth C2f module. The output of the fifth C2f module is the fifth neck feature. The third downsampling fusion unit includes a third downsampling module and a sixth C2f module connected in sequence; the input of the third downsampling fusion unit includes a fifth neck feature and a fifth fusion feature; The fifth neck feature is used as the input to the third downsampling module. The output of the third downsampling module is concatenated with the fifth fusion feature channel and then input to the sixth C2f module. The output of the sixth C2f module is the sixth neck feature.

5. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 1, characterized in that, The detection head module includes a small target detection head, a medium-small target detection head, a medium target detection head, a large target detection head, and a candidate result filtering module; The input to the small target detection head is the third neck feature; the output of the small target detection head is the small target detection result. The input to the small and medium-sized target detection head is the fourth neck feature; the output of the small and medium-sized target detection head is the small and medium-sized target detection result. The input to the medium-sized target detection head is the fifth neck feature; the output of the medium-sized target detection head is the medium-sized target detection result. The input to the large target detection head is the sixth neck feature; The output of the large target detection head is the large target detection result; The large target detection results, medium target detection results, medium-small target detection results, and small target detection results all include target detection boxes, confidence scores, and class probabilities. The candidate result screening module first calculates the product of confidence and category probability in the detection results of large targets, medium targets, medium and small targets, and small targets, and retains the detection results whose product is greater than the preset confidence threshold as candidate detection results; Candidate detection results are sorted from high to low according to the product of confidence and class probability. Non-maximum suppression is then performed based on the cross-union ratio (CUP) between the predicted target detection boxes to remove target detection boxes whose overlap with high-confidence predicted target detection boxes is greater than the preset CUP threshold, thus obtaining the final detection results.

6. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 1, characterized in that, The total loss function L for object detection is: ； in, , , These represent the bounding box regression loss, the category classification loss, and the distribution focus loss, respectively. , and These are the weight coefficients for bounding box regression loss, category classification loss, and distribution focus loss, respectively.

7. The method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement according to claim 1, characterized in that, The preprocessing process is as follows: The original visible light image and the original infrared image at the same observation location and in the same scene are acquired in chronological order. The acquired original visible light image and the original infrared image are paired and subjected to size unification and synchronous data enhancement processing to obtain infrared-visible light image pairs. The synchronous data augmentation process includes one or more of random scaling, random flipping, random cropping, random translation, and Mosaic data augmentation. Synchronous data augmentation applies the same geometric transformation to the target bounding boxes of both visible light and infrared images to ensure that the two modalities maintain their spatial correspondence after augmentation. The target categories include pedestrians, ships, cars, buses, bicycles, airplanes, and drones; the bounding box Normalized coordinates are represented as: ； in, and Indicates the coordinates of the center point of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box.

8. A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes the computer program, it implements a method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements a method for detecting small targets in unmanned aerial vehicles based on cross-modal multi-scale enhancement as described in any one of claims 1-7.

Images