A multi-modal detection model and method for small-scale unmanned aerial vehicles

Abstract

The application specifically relates to a multi-modal detection model and method for small-scale unmanned aerial vehicles, which aims to solve the technical problems that existing detection methods are susceptible to environmental factors, depend on spatial domain feature aggregation, have low recognition rate in complex backgrounds and are prone to losing small target features; the application comprises a visible-infrared dual-flow feature extraction unit, a neck fusion unit and a detection head unit; the neck fusion unit comprises a first DWBF module, a second DWBF module and a third DWBF module, is used for performing frequency domain decomposition on input features, and realizes cross-modal noise suppression and information enhancement through a differentiated reconstruction rule, so that cross-modal interference is suppressed at the source; the detection head unit comprises a first ASCA module and a second ASCA module, the first ASCA module and the second ASCA module are used for performing spatial attention enhancement, effectively stripping weak target signals from complex background clutter, and thus the positioning accuracy of the detection head to small targets is significantly enhanced.

Description

Technical Field

[0001] This invention relates to a small target detection system and method, specifically to a multimodal detection model and method for small-scale unmanned aerial vehicles (UAVs). Background Technology

[0002] With the rapid development of drones, their applications in low-altitude economy, logistics transportation, and emergency rescue are becoming increasingly widespread. However, the proliferation of drones has also brought significant security risks and privacy concerns. Due to their small size, high speed, and complex background environments, drone detection and identification face severe challenges. In real-world scenarios, small-scale drones typically refer to drones or light aircraft that occupy a small pixel area in images, fly at low altitudes, and have blurred target boundaries. These targets often exhibit low resolution, weak texture, and high dynamic characteristics, making them easily obscured by complex backgrounds or confused with noise. Therefore, small-scale drone detection is more challenging than conventional target detection.

[0003] Traditional single-modal detection methods are easily affected by environmental factors in practical applications. Visible light images can effectively provide texture and shape information of UAVs under daytime conditions, but their performance degrades significantly at night or in severe weather conditions. Infrared light images have stable detection capabilities in low-light environments, but due to the lack of detailed information, they are easily affected by heat sources, leading to an increased false detection rate.

[0004] The fusion of visible and infrared images offers a novel approach to improving detection robustness, enabling more stable target perception in diverse environments through modal complementarity. However, existing fusion methods largely rely on spatial domain feature aggregation, neglecting the inherent spectral differences between visible and infrared images. This can easily lead to the coupling of infrared noise with visible textures, affecting the fusion results. Furthermore, small-scale UAV targets typically possess highly dynamic characteristics, and standard convolution operations, limited by local receptive fields, struggle to effectively extract target signals from complex backgrounds, resulting in the suppression of small target features. Therefore, fully leveraging the spectral differences between visible and infrared images while avoiding the loss and interference of small target features to improve detection accuracy and stability remains a critical challenge. Summary of the Invention

[0005] The main objective of this invention is to address the technical problems of existing detection methods, such as susceptibility to environmental interference, reliance on spatial domain feature aggregation, low recognition rate in complex backgrounds, and easy loss of small target features. In response, this invention provides a multimodal detection model and method for small-scale unmanned aerial vehicles (UAVs).

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] A multimodal detection model for small-scale UAVs includes a visible-infrared dual-stream feature extraction unit, a neck fusion unit, and a detection head unit; its special feature is:

[0008] The visible-infrared dual-stream feature extraction unit includes a first feature extraction branch and a second feature extraction branch. The input end of the first feature extraction branch is used to input the original visible light image and perform initial feature extraction on it to obtain a visible light feature map. The second feature extraction branch is used to input the original infrared light image and perform initial feature extraction on it to obtain the infrared light feature map. ;

[0009] The neck fusion unit includes a first DWBF module, a second DWBF module, a third DWBF module, and an SPPF module connected to the output of the third DWBF module; the outputs of the first feature extraction branch and the second feature extraction branch are respectively connected to the inputs of the first DWBF module, the second DWBF module, and the third DWBF module; the first DWBF module, the second DWBF module, and the third DWBF module are respectively used to process visible light feature maps. and infrared light feature map Frequency domain decoupling and fusion are performed, and the global receptive field is enhanced by using the SPPF module to process the results of the third DWBF module, resulting in the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature; the outputs of the first DWBF module, the second DWBF module, and the SPPF module are connected to the detection head unit respectively.

[0010] The detection head unit is used to perform spatial decoupling and spatial attention enhancement on the first multi-scale features, the second multi-scale features, and the third multi-scale features to obtain the final target detection result.

[0011] Furthermore, the first DWBF module, the second DWBF module, and the third DWBF module all include a pre-alignment module, a first wavelet decomposition module, a second wavelet decomposition module, a low-frequency fusion module, a high-frequency fusion module, an inverse wavelet reconstruction module, a residual fusion module, and a feature adaptation module;

[0012] The pre-alignment module includes a first depthwise convolutional layer and a second depthwise convolutional layer, the first depthwise convolutional layer being used to receive visible light feature maps. Channel alignment is performed, and its output is connected to the input of the first wavelet decomposition module; the second deep convolutional layer is used to receive infrared feature maps. It performs channel alignment, and its output is connected to the input of the second wavelet decomposition module;

[0013] The first wavelet decomposition module is used to analyze the visible light feature map. The infrared light feature map is decomposed to obtain the low-frequency sub-band and the high-frequency sub-band of visible light. The second wavelet decomposition module is used to analyze the infrared light feature map. The decomposition process yields a low-frequency subband and a high-frequency subband for infrared light. The outputs of the first and second wavelet decomposition modules are simultaneously connected to the inputs of the low-frequency fusion module and the high-frequency fusion module.

[0014] The low-frequency fusion module is used to fuse the visible light low-frequency sub-band and the infrared light low-frequency sub-band, and the high-frequency fusion module is used to fuse the visible light high-frequency sub-band and the infrared light high-frequency sub-band; the output terminals of the low-frequency fusion module and the high-frequency fusion module are both connected to the input terminal of the inverse wavelet reconstruction module.

[0015] The input and output terminals of the inverse wavelet reconstruction module, residual fusion module, and feature adaptation module are connected in sequence; the residual fusion module includes a learnable intensity parameter γ and a first weighted superposition module, and its other input terminal is connected to the visible light feature map. ;

[0016] The inverse wavelet reconstruction module is used to perform inverse wavelet reconstruction on the low-frequency fusion result and the high-frequency fusion result to obtain the reconstructed fusion feature map; the residual fusion module is used to process the reconstructed fusion feature map using a learnable intensity parameter γ and then combine it with the visible light feature map. Weighted fusion is performed to obtain a fused residual feature map; the feature adaptation module is used to unify the distribution and smooth the features of the fused residual feature map to obtain multi-scale features.

[0017] Furthermore, the detection head unit includes a first upsampling module, a first feature stitching module, a fifth C3k2 module, a second upsampling module, a second feature stitching module, a sixth C3k2 module, a first ASCA module, a first downsampling module, a third feature stitching module, a seventh C3k2 module, a second ASCA module, a second downsampling module, a fourth feature stitching module, and an eighth C3k2 module connected in sequence to input and output; and a first detection module connected to the output of the first ASCA module, a second detection module connected to the output of the second ASCA module, and a third detection module connected to the output of the eighth C3k2 module.

[0018] The input of the first upsampling module is connected to the output of the SPPF module, the other input of the first feature stitching module is connected to the output of the second DWBF module, the other input of the second feature stitching module is connected to the output of the first DWBF module, the other input of the third feature stitching module is connected to the output of the fifth C3k2 module, and the other input of the fourth feature stitching module is connected to the output of the SPPF module.

[0019] Both the first ASCA module and the second ASCA module are used to perform spatial attention enhancement on their input feature maps;

[0020] The first detection module, the second detection module, and the third detection module are used to output the final target detection result.

[0021] Furthermore, both the first ASCA module and the second ASCA module include a global context modeling unit, a horizontal strip convolution unit, a vertical strip convolution unit, a feature fusion unit, an activation map generation unit, and an attention weighting unit; the input ends of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all used to connect to the input feature map;

[0022] The global context modeling unit is used to perform global context expansion on the input feature map to obtain an expanded global context feature map; the horizontal strip convolution unit is used to extract long-distance dependency features in the horizontal direction to obtain a horizontal feature map; the vertical strip convolution unit is used to extract long-distance dependency features in the vertical direction to obtain a vertical feature map.

[0023] The outputs of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all connected to the input of the feature fusion unit.

[0024] The input and output terminals of the feature fusion unit, activation map generation unit, and attention weighting unit are connected in sequence; the other input terminal of the attention weighting unit is used to connect to the input feature map; the feature fusion unit is used to perform element-wise addition operations on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map; the activation map generation unit is used to process the fused feature map to generate a spatial attention map; the attention weighting unit is used to perform element-wise multiplication operations on the input feature map and the spatial attention map;

[0025] The global context modeling unit includes an adaptive average pooling layer, a fourth 1×1 convolutional layer, and a third batch normalization layer, with the input and output connected in sequence; the horizontal strip convolutional unit and the vertical strip convolutional unit each include a 1×11 convolutional layer.

[0026] Furthermore, both the first feature extraction branch and the second feature extraction branch include a first convolutional layer, a second convolutional layer, a first C3k2 module, a third convolutional layer, a second C3k2 module, a fourth convolutional layer, a third C3k2 module, a fifth convolutional layer, and a fourth C3k2 module, which are connected in sequence from input to output.

[0027] In the first and second feature extraction branches, the output of the second C3k2 module is connected to the two inputs of the first DWBF module, respectively.

[0028] The output of the third C3k2 module in the first and second feature extraction branches is connected to the two inputs of the second DWBF module, respectively.

[0029] The output of the fourth C3k2 module in the first and second feature extraction branches is connected to the two inputs of the third DWBF module, respectively.

[0030] Furthermore, the low-frequency fusion module includes a spatial weighted generation unit and a weighted fusion unit with inputs and outputs connected in sequence; wherein, the spatial weighted generation unit includes a feature splicing layer, a first 1×1 convolutional layer, a first batch normalization layer, a ReLU activation layer, a second 1×1 convolutional layer, and a Softmax layer with inputs and outputs connected in sequence, for outputting weights α and β; the weighted fusion unit includes a first weighted channel, a second weighted channel, and a low-frequency fusion unit connected to the output of the first weighted channel and the output of the second weighted channel;

[0031] The high-frequency fusion module includes a physical rule fusion unit, a feature splicing module, a channel attention module, an element-wise multiplication module, and a high-frequency subband splitting module Split, with the input and output connected in sequence; wherein, the physical rule fusion unit includes a first selection module Max; the channel attention module includes a global average pooling layer, a convolutional layer, and an activation function Sigmoid, with the input and output connected in sequence.

[0032] The feature adaptation module includes a third 1×1 convolutional layer, a second batch normalization layer, and a SiLu activation layer, and its input is connected to the output of the residual fusion module.

[0033] Meanwhile, the present invention also provides a multimodal detection method for small-scale unmanned aerial vehicles, comprising the following steps:

[0034] Step 1: Construct the multimodal detection model for small-scale UAVs as described above;

[0035] Step 2: Acquire a raw visible light image and a corresponding raw infrared light image to be detected;

[0036] Step 3: Input the original visible light image and the original infrared light image into the model; use the first feature extraction branch in the visible-infrared dual-stream feature extraction unit to perform initial feature extraction on the original visible light image to obtain the visible light feature map. The second feature extraction branch in the visible-infrared dual-stream feature extraction unit is used to perform initial feature extraction on the original infrared light image to obtain an infrared light feature map. ;

[0037] Step 4: Use the first DWBF module, the second DWBF module, and the third DWBF module in the neck fusion unit to process the visible light feature map respectively. and infrared light feature map Frequency domain decoupling and fusion are performed, and the global receptive field is enhanced using the SPPF module to obtain the first multi-scale feature, the second multi-scale feature and the third multi-scale feature.

[0038] Step 5: Use the detection head unit to perform spatial decoupling and spatial attention enhancement on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature respectively to obtain the final target detection result and complete the UAV detection.

[0039] Furthermore, in step 4, the first DWBF module in the neck fusion unit is used to process the visible light feature map. and infrared light feature map The specific steps for frequency domain decoupling and fusion are as follows:

[0040] S1. Use the first depthwise convolutional layer in the pre-alignment module of the first DWBF module to process the visible light feature map. Channel alignment is performed, and the signal is then input into the first wavelet decomposition module for decomposition to obtain the visible light low-frequency subband. and visible light high-frequency subband , , ;

[0041] The infrared feature map is processed using the second deep convolutional layer in the pre-aligned module. Channel alignment is performed, and the signal is then input into the second wavelet decomposition module for decomposition to obtain the infrared low-frequency subband. Infrared high-frequency subband , , ;

[0042] S2. Using a low-frequency fusion module to optimize the low-frequency subband of visible light. and infrared low-frequency subband The fusion process yields low-frequency fusion results. ;

[0043] S2.1 Using spatially weighted generation units for visible light low-frequency subband and infrared low-frequency subband The weights α and β are obtained through processing.

[0044] S2.2, Use the first weighted channel in the weighted fusion unit to perform fusion on the visible light low-frequency subband. The first fusion result is obtained by fusing the weighted signal with weight α; the second weighted channel is used to analyze the low-frequency subband of the infrared light. The first fusion result is obtained by fusing the first fusion result with the weight β; the second fusion result is then fused using a low-frequency fusion unit to obtain the low-frequency fusion result. ;

[0045] S3. Using a high-frequency fusion module to optimize visible light high-frequency subbands. , , and infrared high-frequency subband , , The fusion was performed to obtain high-frequency fusion results. , and ;

[0046] S3.1 Using physical rule fusion units for visible light high-frequency subband , , and infrared high-frequency subband , , Perform maximum absolute value selection fusion to obtain the selection fusion result. , and ;

[0047] S3.2, Use the feature stitching module to select the fusion results. , and After fusion, the input is processed into the channel attention module to generate a channel attention map;

[0048] S3.3. Use the element-wise multiplication module to fuse the channel attention map with the selection result. , , The high-frequency feature map is obtained by fusing the features. ;

[0049] S3.4. Use the high-frequency subband splitting module Split to split the fused high-frequency feature map. Decomposed into high-frequency fusion results , and ;

[0050] S4. Use the inverse wavelet reconstruction module to fuse the low-frequency results. High-frequency fusion results , and Perform inverse wavelet reconstruction to obtain the fused feature map. ;

[0051] S5. Use the learnable intensity parameter γ in the residual fusion module to fuse the feature map. The data is processed and then compared with the visible light feature map. The samples are input into the first weighted overlay module for fusion to obtain the fused residual feature map;

[0052] S6. Use the feature adaptation module to perform distribution unification and feature smoothing on the fused residual feature map to obtain the first multi-scale feature.

[0053] Furthermore, step 5 specifically includes:

[0054] Step 5.1: Use the first upsampling module in the detection head unit to upsample the third multi-scale feature to obtain feature map U1; use the first feature stitching module to stitch feature map U1 and the second multi-scale feature to obtain feature map C1; use the fifth C3k2 module to process feature map C1 to obtain feature map F1.

[0055] Step 5.2: Upsample feature map F1 using the second upsampling module to obtain feature map U2; stitch feature map U2 and the first multi-scale feature using the second feature stitching module to obtain feature map C2; process feature map C2 using the sixth C3k2 module to obtain feature map F2.

[0056] Step 5.3: Use the first ASCA module to perform spatial attention enhancement on feature map F2 to obtain feature map A2; output feature map A2 through the first detection module;

[0057] Step 5.4: Use the first downsampling module to downsample feature map A2 to obtain feature map D1; use the third feature concatenation module to concatenate feature map D1 and feature map F1 along the channel dimension to obtain feature map C3; use the seventh C3k2 module to process C3 to obtain feature map F3.

[0058] Step 5.5: Use the second ASCA module to perform spatial attention enhancement on feature map F3 to obtain feature map A1; output feature map A1 through the second detection module;

[0059] Step 5.6: Use the second downsampling module to downsample feature map A1 to obtain feature map D2; use the fourth feature stitching module to stitch D2 and the third multi-scale feature along the channel dimension to obtain feature map C4; use the eighth C3k2 module to process C4 to obtain feature map F4; output feature map F4 through the third detection module.

[0060] The feature maps A1, A2, and F4 are the final detection results.

[0061] Furthermore, step 5.3 specifically includes:

[0062] Step 5.3.1: Input feature map F2 into the global context modeling unit of the first ASCA module to obtain the global context feature map; expand the global context feature map through a broadcast mechanism to obtain the expanded global context feature map;

[0063] Step 5.3.2: Input feature map F2 into the horizontal strip convolutional unit of the first ASCA module to extract long-distance dependency features in the horizontal direction and obtain the horizontal feature map;

[0064] Step 5.3.3: Input feature map F2 into the vertical strip convolutional unit of the first ASCA module to extract long-range dependent features in the vertical direction and obtain the vertical feature map.

[0065] Step 5.3.4: Use the feature fusion unit to perform element-wise addition on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map;

[0066] Step 5.3.5: Use the activation mapping generation unit to process the fused feature map into a spatial attention map;

[0067] Step 5.3.6: Perform element-wise multiplication on feature map F2 and spatial attention map using the attention weighting unit to generate feature map A2; output feature map A2 through the first detection module;

[0068] The size of the global context feature map is (B, C, 1, 1); the sizes of the expanded global context feature map, horizontal feature map, vertical feature map, fused feature map, and spatial attention map are all (B, C, H, W).

[0069] Compared with the prior art, the beneficial effects of the present invention are:

[0070] 1. The present invention provides a multimodal detection model and method for small-scale UAVs. By combining a visible-infrared dual-stream feature extraction unit, a neck fusion unit, and a detection head unit, the UAV target image is detected. It can make full use of the spectral differences between visible light images and infrared light images under different modes to avoid the suppression of small target features and improve the accuracy of detection results.

[0071] 2. This invention provides a multimodal detection model and method for small-scale UAVs. It employs a DWBF (Discrete Wavelet Band-Differentiated Fusion) module to fuse features from visible light and infrared images, using discrete wavelet transform to explicitly decompose the features. This enables the network to separate background thermal noise from effective texture information in high-frequency components and fuse them through differentiated reconstruction rules, thereby suppressing cross-modal interference at its source.

[0072] 3. This invention provides a multimodal detection model and method for small-scale UAVs, which uses the ASCA (Axial Strip Contextual Attention) module to establish long-range dependencies through anisotropic strip convolution. It can generate a sharp spatial attention map, effectively separating weak target signals from complex background clutter, and significantly improving the localization accuracy of the detection head for small targets. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the structure of an embodiment of a multimodal detection model for small-scale unmanned aerial vehicles according to the present invention;

[0074] Figure 2 This is a schematic diagram of the structure of the first DWBF module in an embodiment of a multimodal detection model for small-scale unmanned aerial vehicles according to the present invention;

[0075] Figure 3 This is a schematic diagram of the structure of the first ASCA module in an embodiment of a multimodal detection model for small-scale unmanned aerial vehicles according to the present invention;

[0076] Figure 4 This is a comparison chart showing the results of validation using the Anti-UAV300 dataset between the multimodal detection method for small-scale UAVs proposed in this invention and traditional methods.

[0077] Figure 5 This is a comparison chart showing the results of validation using the MMFW-UAV dataset with the multimodal detection method for small-scale UAVs proposed in this invention and traditional methods. Detailed Implementation

[0078] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0079] A multimodal detection model for small-scale UAVs, the specific structure of which is as follows: Figure 1 As shown, it includes a visible infrared dual-stream feature extraction unit, a neck fusion unit, and a detection head unit.

[0080] The visible-infrared dual-stream feature extraction unit includes a first feature extraction branch and a second feature extraction branch. The input of the first feature extraction branch is used to input the original visible light image and perform initial feature extraction on it to obtain a visible light feature map. The second feature extraction branch is used to input the original infrared light image and perform initial feature extraction on it, maintaining modal independence to obtain an infrared light feature map. The first and second feature extraction branches each include a first convolutional layer, a second convolutional layer, a first C3k2 module, a third convolutional layer, a second C3k2 module, a fourth convolutional layer, a third C3k2 module, a fifth convolutional layer, and a fourth C3k2 module, all connected sequentially to the input and output. The kernel size of the first, second, third, fourth, and fifth convolutional layers is 3×3, with a stride of 2, used to extract spatial texture features step by step.

[0081] In this embodiment, the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer are all Conv.

[0082] The neck fusion unit includes a first DWBF module, a second DWBF module, a third DWBF module, and an SPPF module connected to the output of the third DWBF module. The outputs of the first and second feature extraction branches are connected to the inputs of the first, second, and third DWBF modules, respectively. The two inputs of the first DWBF module are connected to the outputs of the two second C3k2 modules, respectively. The two inputs of the second DWBF module are connected to the outputs of the two third C3k2 modules, respectively. The two inputs of the third DWBF module are connected to the outputs of the two fourth C3k2 modules, respectively. The SPPF (Spatial Pyramid Pooling Fast) module is used to expand the receptive field through multi-scale pooling operations, thereby improving the global expressive power of the fused features.

[0083] The first, second, and third DWBF modules are used to perform frequency domain decomposition on the input features and to achieve cross-modal noise suppression and information enhancement through differential reconstruction rules. All three modules have the same structure. The specific structure of the first DWBF module is as follows: Figure 2 As shown, it includes a pre-alignment module, a first wavelet decomposition module, a second wavelet decomposition module, a low-frequency fusion module, a high-frequency fusion module, an inverse wavelet reconstruction module, a residual fusion module, and a feature adaptation module.

[0084] The pre-alignment module includes a first depthwise convolutional layer and a second depthwise convolutional layer. The first depthwise convolutional layer is used to receive visible light feature maps. Channel alignment is performed, and its output is connected to the input of the first wavelet decomposition module; the second deep convolutional layer is used to receive infrared feature maps. It performs channel alignment, and its output is connected to the input of the second wavelet decomposition module.

[0085] The first wavelet decomposition module is used to analyze the visible light feature map. The infrared light feature map is decomposed to obtain the low-frequency sub-band and the high-frequency sub-band of visible light. The second wavelet decomposition module is used to analyze the infrared light feature map. The wavelet decomposition process yields a low-frequency subband and a high-frequency subband for infrared light. The outputs of the first and second wavelet decomposition modules are simultaneously connected to the inputs of the low-frequency fusion module and the high-frequency fusion module. The low-frequency fusion module is used to fuse the low-frequency subband for visible light and the low-frequency subband for infrared light, while the high-frequency fusion module is used to fuse the high-frequency subband for visible light and the high-frequency subband for infrared light. The outputs of both the low-frequency and high-frequency fusion modules are connected to the input of the inverse wavelet reconstruction module.

[0086] The input and output terminals of the inverse wavelet reconstruction module, the residual fusion module, and the feature adaptation module are connected sequentially; the residual fusion module includes a learnable intensity parameter γ and a first weighted superposition module, and its other input terminal is connected to the visible light feature map. Where γ is a scalar parameter, set to 0.1 in this embodiment, and can be dynamically scheduled during training; the inverse wavelet reconstruction module is used to perform inverse wavelet reconstruction on the low-frequency fusion result and the high-frequency fusion result to obtain the reconstructed fusion feature map; the residual fusion module is used to process the reconstructed fusion feature map through the learnable intensity parameter γ and then combine it with the visible light feature map. Weighted fusion is performed to obtain a fused residual feature map; the feature adaptation module is used to unify the distribution and smooth the features of the fused residual feature map to obtain multi-scale features.

[0087] In this embodiment, the pre-alignment module is DWConv (Depthwise Convolution), the first wavelet decomposition module and the second wavelet decomposition module are both DWT (Discrete Wavelet Transform), and the inverse wavelet reconstruction module is IDWT (Inverse Discrete Wavelet Transform).

[0088] Specifically, the low-frequency fusion module includes a spatially weighted generation unit and a weighted fusion unit connected sequentially by input and output. The spatially weighted generation unit includes a feature concatenation module, a first 1×1 convolutional layer, a first batch normalization layer, a ReLU activation layer, a second 1×1 convolutional layer, and a Softmax layer connected sequentially by input and output, used to output weights α and β. The weighted fusion unit includes a first weighted channel, a second weighted channel, and a low-frequency fusion unit connected to the outputs of the first and second weighted channels. The high-frequency fusion module includes a physical rule fusion unit, a feature concatenation module, a channel attention module, an element-wise multiplication module, and a high-frequency subband splitting module (Split) connected sequentially by input and output. The physical rule fusion unit includes a first selection module (Max). The channel attention module includes a global average pooling layer, a convolutional layer, and a Sigmoid activation function connected sequentially by input and output. The feature adaptation module includes a third 1×1 convolutional layer, a second batch normalization layer, and a SiLU activation layer, with its input connected to the output of the residual fusion module.

[0089] In this embodiment, the first 1×1 convolutional layer, the first batch normalization layer, the ReLU activation layer, and the second 1×1 convolutional layer are connected sequentially as Conv, BN (Batch Normalization), ReLU (Rectified Linear Unit), and Conv, respectively; the first weighting channel is an element-wise multiplication module that receives weight α, the second weighting channel is an element-wise multiplication module that receives weight β, and the low-frequency fusion unit is an element-wise addition module.

[0090] The global average pooling layer is GAP (Global Average Pooling), the convolutional layer is Conv, and the third 1×1 convolutional layer and the second batch normalization layer are BN.

[0091] The input of the detection head unit is connected to the output of the neck fusion unit to receive fused features; its output is used to output the target detection results, including the target category and bounding box. The detection head unit includes a first upsampling module, a first feature stitching module, a fifth C3k2 module, a second upsampling module, a second feature stitching module, a sixth C3k2 module, a first ASCA module, a first downsampling module, a third feature stitching module, a seventh C3k2 module, a second ASCA module, a second downsampling module, a fourth feature stitching module, an eighth C3k2 module, and a module connected in sequence to the input and output. The module output is connected to a first detection module, a second detection module connected to the output of a second ASCA module, and a third detection module connected to the output of an eighth C3k2 module; wherein, the input of the first upsampling module is connected to the output of the SPPF module, another input of the first feature stitching module is connected to the output of the second DWBF module, another input of the second feature stitching module is connected to the output of the first DWBF module, another input of the third feature stitching module is connected to the output of the fifth C3k2 module, and another input of the fourth feature stitching module is connected to the output of the SPPF module.

[0092] The first ASCA module and the second ASCA module have the same structure and are both used to generate spatial attention maps to enhance the localization accuracy of small targets. The first C3k2 module, the second C3k2 module, the third C3k2 module and the fourth C3k2 module are used to improve feature representation capabilities. The first detection module, the second detection module and the third detection module are all multi-scale detection heads that can simultaneously output the detection results of targets at different scales, including target categories and bounding boxes.

[0093] In this embodiment, both the first upsampling module and the second upsampling module are Upsample; both the first downsampling module and the second downsampling module are Conv; and both the first detection module, the second detection module, and the third detection module are Detect.

[0094] Specifically, such as Figure 3 As shown, the first ASCA module includes a global context modeling unit, a horizontal strip convolution unit, a vertical strip convolution unit, a feature fusion unit, an activation map generation unit, and an attention weighting unit; the input ends of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all used to connect to the input feature map.

[0095] The global context modeling unit is used to extend the input feature map with global context to obtain the extended global context feature map; the horizontal strip convolution unit is used to extract long-range dependency features in the horizontal direction to obtain the horizontal feature map; the vertical strip convolution unit is used to extract long-range dependency features in the vertical direction to obtain the vertical feature map.

[0096] The outputs of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all connected to the input of the feature fusion unit; the inputs and outputs of the feature fusion unit, the activation map generation unit, and the attention weighting unit are connected in sequence; the other input of the attention weighting unit is used to connect to the input feature map.

[0097] The feature fusion unit performs element-wise addition on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map; the activation map generation unit processes the fused feature map to generate a spatial attention map; and the attention weighting unit performs element-wise multiplication on the input feature map and the spatial attention map.

[0098] In this embodiment, the feature fusion unit is an element-wise addition module, the activation map generation unit is a Sigmoid module, and the attention weighting unit is an element-wise multiplication module.

[0099] The global context modeling unit includes a first adaptive average pooling layer, a fourth 1×1 convolutional layer, and a third batch normalization layer, which are connected sequentially to provide semantic-level background awareness. The horizontal and vertical strip convolutional units each include a 1×11 convolutional layer, which captures long-distance dependencies through anisotropic convolutional kernels. The feature fusion unit and the activation map generation unit jointly generate a spatial attention map, which effectively enhances the response intensity of small target regions. The attention weighting unit applies the spatial attention map to the original features to achieve spatial decoupling between the target and the background.

[0100] In this embodiment, the first adaptive average pooling layer is Adaptive AvgPool2d(1×1), and the 1×11 convolutional layer is Conv2d(1×11, DW).

[0101] This embodiment also provides a multimodal detection method for small-scale unmanned aerial vehicles, including the following steps:

[0102] Step 1: Construct the multimodal detection model for small-scale UAVs as described above;

[0103] Step 2: Acquire a raw visible light image and a corresponding raw infrared light image to be detected;

[0104] Step 3: Input the original visible light image and the original infrared light image into the model; use the first feature extraction branch in the visible-infrared dual-stream feature extraction unit to perform initial feature extraction on the original visible light image to obtain the visible light feature map. The second feature extraction branch in the visible-infrared dual-stream feature extraction unit is used to perform initial feature extraction on the original infrared light image to obtain an infrared light feature map. ;

[0105] Step 4: Use the first DWBF module, the second DWBF module, and the third DWBF module in the neck fusion unit to process the visible light feature map respectively. and infrared light feature map Frequency domain decoupling and fusion are performed, and the global receptive field is enhanced using the SPPF module to obtain the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature.

[0106] The following process uses the first DWBF module as an example to illustrate the frequency domain decoupling and fusion process:

[0107] S1. Using the first depthwise convolutional layer in the pre-alignment module of the first DWBF module, the visible light feature map is processed by depthwise convolution. Channel alignment is performed, and the signal is then input into the first wavelet decomposition module for decomposition to obtain the visible light low-frequency subband. and visible light high-frequency subband , , ;

[0108] Using the second depthwise convolutional layer in the pre-alignment module, the infrared feature map is processed by depthwise convolution. Channel alignment is performed, and the signal is then input into the second wavelet decomposition module for decomposition to obtain the infrared low-frequency subband. Infrared high-frequency subband , , ;

[0109] S2. Using a low-frequency fusion module to optimize the low-frequency subband of visible light. and infrared low-frequency subband The fusion process yields low-frequency fusion results. ;

[0110] S2.1 Using spatially weighted generation units for visible light low-frequency subband and infrared low-frequency subband The weights α and β are obtained through processing.

[0111] S2.2, Use the first weighted channel in the weighted fusion unit to perform fusion on the visible light low-frequency subband. The first fusion result is obtained by fusing the weighted signal with weight α; the second weighted channel is used to analyze the low-frequency subband of the infrared light. The first fusion result is obtained by fusing the first fusion result with the weight β; the second fusion result is then fused using a low-frequency fusion unit to obtain the low-frequency fusion result. ;

[0112] S3. Using a high-frequency fusion module to optimize visible light high-frequency subbands. , , and infrared high-frequency subband , , The fusion was performed to obtain high-frequency fusion results. , and ;

[0113] S3.1 Using physical rule fusion units for visible light high-frequency subband , , and infrared high-frequency subband , , Perform maximum absolute value selection fusion to obtain the selection fusion result. , and ;

[0114] S3.2, Use the feature stitching module to select the fusion results. , and After fusion, the input is processed into the channel attention module to generate a channel attention map;

[0115] S3.3. Use the element-wise multiplication module to fuse the channel attention map with the selection result. , , The high-frequency feature map is obtained by fusing the features. ;

[0116] S3.4. Use the high-frequency subband splitting module Split to split the fused high-frequency feature map. Decomposed into high-frequency fusion results , and ;

[0117] S4. Use the inverse wavelet reconstruction module to fuse the low-frequency results. High-frequency fusion results , and Perform inverse wavelet reconstruction to obtain the fused feature map. ;

[0118] S5. Use the learnable intensity parameter γ in the residual fusion module to fuse the feature map. The data is processed and then compared with the visible light feature map. The samples are input into the first weighted overlay module for fusion to obtain the fused residual feature map;

[0119] S6. Use the feature adaptation module to perform distribution unification and feature smoothing on the fused residual feature map to obtain the first multi-scale feature.

[0120] Step 5: Use the detection head unit to perform spatial decoupling and spatial attention enhancement on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature respectively to obtain the final target detection result and complete the UAV detection;

[0121] Step 5.1: Use the first upsampling module in the detection head unit to upsample the third multi-scale feature to obtain feature map U1; use the first feature stitching module to stitch feature map U1 and the second multi-scale feature to obtain feature map C1; use the fifth C3k2 module to process feature map C1 to obtain feature map F1.

[0122] Step 5.2: Upsample feature map F1 using the second upsampling module to obtain feature map U2; stitch feature map U2 and the first multi-scale feature using the second feature stitching module to obtain feature map C2; process feature map C2 using the sixth C3k2 module to obtain feature map F2.

[0123] Step 5.3: Use the first ASCA module to perform spatial attention enhancement on feature map F2 to obtain feature map A2; output feature map A2 through the first detection module;

[0124] Step 5.3.1: Input feature map F2 into the global context modeling unit of the first ASCA module to obtain the global context feature map; expand the global context feature map through a broadcast mechanism to obtain the expanded global context feature map;

[0125] Step 5.3.2: Input feature map F2 into the horizontal strip convolutional unit of the first ASCA module to extract long-distance dependency features in the horizontal direction and obtain the horizontal feature map;

[0126] Step 5.3.3: Input feature map F2 into the vertical strip convolutional unit of the first ASCA module to extract long-range dependent features in the vertical direction and obtain the vertical feature map.

[0127] Step 5.3.4: Use the feature fusion unit to perform element-wise addition on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map;

[0128] Step 5.3.5: Use the activation mapping generation unit to process the fused feature map into a spatial attention map;

[0129] Step 5.3.6: Perform element-wise multiplication on feature map F2 and spatial attention map using the attention weighting unit to generate feature map A2; output feature map A2 through the first detection module;

[0130] Step 5.4: Use the first downsampling module to downsample feature map A2 to obtain feature map D1; use the third feature concatenation module to concatenate feature map D1 and feature map F1 along the channel dimension to obtain feature map C3; use the seventh C3k2 module to process C3 to obtain feature map F3.

[0131] Step 5.5: Use the second ASCA module to perform spatial attention enhancement on feature map F3 to obtain feature map A1; output feature map A1 through the second detection module; the specific steps of this process are the same as those in step 5.3, and will not be elaborated here.

[0132] Step 5.6: Use the second downsampling module to downsample feature map A1 to obtain feature map D2; use the fourth feature stitching module to stitch D2 and the third multi-scale feature along the channel dimension to obtain feature map C4; use the eighth C3k2 module to process C4 to obtain feature map F4; output feature map F4 through the third detection module.

[0133] The global context feature map has a size of (B, C, 1, 1); the expanded global context feature map, horizontal feature map, vertical feature map, fused feature map, and spatial attention map all have a size of (B, C, H, W); feature map A1, feature map A2, and feature map F4 are the final detection results.

[0134] To verify the effectiveness of this embodiment, mAP50-95 (mean Average Precision) was selected as the evaluation metric. The detection method of this invention was compared with traditional YOLOv8, YOLOv11, YOLOv12, YOLOv13, RT-DETR, and YOLOX methods on two public datasets, Anti-UAV300 and MMFW-UAV. The comparison results are shown in Table 1.

[0135] Table 1 Experimental Results

[0136]

[0137] Generally, a higher mAP50-95 is better, and the visualization results are as follows: Figure 4 and Figure 5 As shown; where, Figure 4 In the image, A-RGB and A-IR, B-RGB and B-IR, and C-RGB and C-IR are three sets of corresponding original visible light images and original infrared light images selected from the public dataset Anti-UAV300, respectively. Figure 5In this context, A-RGB and A-IR, B-RGB and B-IR, and C-RGB and C-IR are three sets of corresponding original visible light images and original infrared light images selected from the public dataset MMFW-UAV, respectively; Ours refers to the method of this invention.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. For those skilled in the art, modifications can be made to the specific technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions protected by the present invention.

Claims

1. A multimodal detection system for small-scale unmanned aerial vehicles (UAVs), comprising a visible-infrared dual-stream feature extraction unit, a neck fusion unit, and a detection head unit; characterized in that: The visible-infrared dual-stream feature extraction unit includes a first feature extraction branch and a second feature extraction branch. The input end of the first feature extraction branch is used to input the original visible light image and perform initial feature extraction on it to obtain a visible light feature map. ; The second feature extraction branch is used to input the original infrared light image and perform initial feature extraction on it to obtain an infrared light feature map. ; The neck fusion unit includes a first DWBF module, a second DWBF module, a third DWBF module, and an SPPF module connected to the output of the third DWBF module; the outputs of the first feature extraction branch and the second feature extraction branch are respectively connected to the inputs of the first DWBF module, the second DWBF module, and the third DWBF module; the first DWBF module, the second DWBF module, and the third DWBF module are respectively used to process visible light feature maps. and infrared light feature map Frequency domain decoupling and fusion are performed, and the global receptive field is enhanced by using the SPPF module to process the results of the third DWBF module, resulting in the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature; the outputs of the first DWBF module, the second DWBF module, and the SPPF module are connected to the detection head unit respectively. The first DWBF module, the second DWBF module, and the third DWBF module all include a pre-alignment module, a first wavelet decomposition module, a second wavelet decomposition module, a low-frequency fusion module, a high-frequency fusion module, an inverse wavelet reconstruction module, a residual fusion module, and a feature adaptation module; The pre-alignment module includes a first depthwise convolutional layer and a second depthwise convolutional layer, the first depthwise convolutional layer being used to receive visible light feature maps. It performs channel alignment, and its output is connected to the input of the first wavelet decomposition module; The second deep convolutional layer is used to receive infrared light feature maps. It performs channel alignment, and its output is connected to the input of the second wavelet decomposition module; The first wavelet decomposition module is used to analyze the visible light feature map. The infrared light feature map is decomposed to obtain the low-frequency sub-band and the high-frequency sub-band of visible light. The second wavelet decomposition module is used to analyze the infrared light feature map. The decomposition process yields a low-frequency subband and a high-frequency subband for infrared light. The outputs of the first and second wavelet decomposition modules are simultaneously connected to the inputs of the low-frequency fusion module and the high-frequency fusion module. The low-frequency fusion module is used to fuse the visible light low-frequency sub-band and the infrared light low-frequency sub-band, and the high-frequency fusion module is used to fuse the visible light high-frequency sub-band and the infrared light high-frequency sub-band; the output terminals of the low-frequency fusion module and the high-frequency fusion module are both connected to the input terminal of the inverse wavelet reconstruction module. The input and output terminals of the inverse wavelet reconstruction module, residual fusion module, and feature adaptation module are connected in sequence; the residual fusion module includes a learnable intensity parameter γ and a first weighted superposition module, and its other input terminal is connected to the visible light feature map. ; The inverse wavelet reconstruction module is used to perform inverse wavelet reconstruction on the low-frequency fusion result and the high-frequency fusion result to obtain the reconstructed fusion feature map; the residual fusion module is used to process the reconstructed fusion feature map using a learnable intensity parameter γ and then combine it with the visible light feature map. Weighted fusion is performed to obtain a fused residual feature map; the feature adaptation module is used to unify the distribution and smooth the features of the fused residual feature map to obtain multi-scale features. The detection head unit is used to perform spatial decoupling and spatial attention enhancement on the first multi-scale features, the second multi-scale features, and the third multi-scale features to obtain the final target detection result.

2. The multimodal detection system for small-scale unmanned aerial vehicles according to claim 1, characterized in that: The detection head unit includes a first upsampling module, a first feature stitching module, a fifth C3k2 module, a second upsampling module, a second feature stitching module, a sixth C3k2 module, a first ASCA module, a first downsampling module, a third feature stitching module, a seventh C3k2 module, a second ASCA module, a second downsampling module, a fourth feature stitching module, and an eighth C3k2 module connected in sequence to input and output; and a first detection module connected to the output of the first ASCA module, a second detection module connected to the output of the second ASCA module, and a third detection module connected to the output of the eighth C3k2 module. The input of the first upsampling module is connected to the output of the SPPF module, the other input of the first feature stitching module is connected to the output of the second DWBF module, the other input of the second feature stitching module is connected to the output of the first DWBF module, the other input of the third feature stitching module is connected to the output of the fifth C3k2 module, and the other input of the fourth feature stitching module is connected to the output of the SPPF module. Both the first ASCA module and the second ASCA module include a global context modeling unit, a horizontal strip convolution unit, a vertical strip convolution unit, a feature fusion unit, an activation map generation unit, and an attention weighting unit; the input ends of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all used to connect to the input feature map; The global context modeling unit is used to perform global context expansion on the input feature map to obtain an expanded global context feature map; the horizontal strip convolution unit is used to extract long-distance dependency features in the horizontal direction to obtain a horizontal feature map; the vertical strip convolution unit is used to extract long-distance dependency features in the vertical direction to obtain a vertical feature map. The outputs of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all connected to the input of the feature fusion unit. The input and output terminals of the feature fusion unit, activation map generation unit, and attention weighting unit are connected in sequence; the other input terminal of the attention weighting unit is used to connect to the input feature map. The feature fusion unit is used to perform element-wise addition on the horizontal feature map, the vertical feature map, and the expanded global context feature map to generate a fused feature map. The activation mapping generation unit is used to generate a spatial attention map after processing the fused feature map; the attention weighting unit is used to perform element-wise multiplication operations on the input feature map and the spatial attention map. The global context modeling unit includes an adaptive average pooling layer, a fourth 1×1 convolutional layer, and a third batch normalization layer, with the input and output connected in sequence; the horizontal strip convolutional unit and the vertical strip convolutional unit each include a 1×11 convolutional layer. Both the first ASCA module and the second ASCA module are used to perform spatial attention enhancement on their input feature maps; The first detection module, the second detection module, and the third detection module are used to output the final target detection result.

3. A multimodal detection system for small-scale unmanned aerial vehicles according to claim 1 or 2, characterized in that: The first feature extraction branch and the second feature extraction branch both include a first convolutional layer, a second convolutional layer, a first C3k2 module, a third convolutional layer, a second C3k2 module, a fourth convolutional layer, a third C3k2 module, a fifth convolutional layer, and a fourth C3k2 module, which are connected in sequence from input to output. In the first and second feature extraction branches, the output of the second C3k2 module is connected to the two inputs of the first DWBF module, respectively. The output of the third C3k2 module in the first feature extraction branch and the second feature extraction branch is connected to the two inputs of the second DWBF module, respectively. The output of the fourth C3k2 module in the first and second feature extraction branches is connected to the two inputs of the third DWBF module, respectively.

4. The multimodal detection system for small-scale unmanned aerial vehicles according to claim 1, characterized in that: The low-frequency fusion module includes a spatial weighted generation unit and a weighted fusion unit with inputs and outputs connected in sequence. The spatial weighted generation unit includes a feature splicing layer, a first 1×1 convolutional layer, a first batch normalization layer, a ReLU activation layer, a second 1×1 convolutional layer, and a Softmax layer with inputs and outputs connected in sequence, for outputting weights α and β. The weighted fusion unit includes a first weighted channel, a second weighted channel, and a low-frequency fusion unit connected to the output of the first weighted channel and the output of the second weighted channel. The high-frequency fusion module includes a physical rule fusion unit, a feature splicing module, a channel attention module, an element-wise multiplication module, and a high-frequency subband splitting module Split, with the input and output connected in sequence; wherein, the physical rule fusion unit includes a first selection module Max; the channel attention module includes a global average pooling layer, a convolutional layer, and an activation function Sigmoid, with the input and output connected in sequence. The feature adaptation module includes a third 1×1 convolutional layer, a second batch normalization layer, and a SiLu activation layer, and its input is connected to the output of the residual fusion module.

5. A multimodal detection method for small-scale unmanned aerial vehicles, characterized in that, Includes the following steps: Step 1: Construct a multimodal detection system for small-scale unmanned aerial vehicles as described in any one of claims 1-4; Step 2: Acquire a raw visible light image and a corresponding raw infrared light image to be detected; Step 3: Input the original visible light image and the original infrared light image into the system; use the first feature extraction branch in the visible-infrared dual-stream feature extraction unit to perform initial feature extraction on the original visible light image to obtain the visible light feature map. The second feature extraction branch in the visible-infrared dual-stream feature extraction unit is used to perform initial feature extraction on the original infrared light image to obtain an infrared light feature map. ; Step 4: Use the first DWBF module, the second DWBF module, and the third DWBF module in the neck fusion unit to process the visible light feature map respectively. and infrared light feature map Frequency domain decoupling and fusion are performed, and the global receptive field is enhanced using the SPPF module to obtain the first multi-scale feature, the second multi-scale feature and the third multi-scale feature. The first DWBF module, the second DWBF module, and the third DWBF module each include a pre-alignment module, a first wavelet decomposition module, a second wavelet decomposition module, a low-frequency fusion module, a high-frequency fusion module, an inverse wavelet reconstruction module, a residual fusion module, and a feature adaptation module. The pre-alignment module includes a first depthwise convolutional layer and a second depthwise convolutional layer, the first depthwise convolutional layer being used to receive visible light feature maps. Channel alignment is performed, and its output is connected to the input of the first wavelet decomposition module; the second deep convolutional layer is used to receive infrared feature maps. It performs channel alignment, and its output is connected to the input of the second wavelet decomposition module; The first wavelet decomposition module is used to analyze the visible light feature map. The infrared light feature map is decomposed to obtain the low-frequency sub-band and the high-frequency sub-band of visible light. The second wavelet decomposition module is used to analyze the infrared light feature map. The decomposition process yields a low-frequency subband and a high-frequency subband for infrared light. The outputs of the first and second wavelet decomposition modules are simultaneously connected to the inputs of the low-frequency fusion module and the high-frequency fusion module. The low-frequency fusion module is used to fuse the visible light low-frequency sub-band and the infrared light low-frequency sub-band, and the high-frequency fusion module is used to fuse the visible light high-frequency sub-band and the infrared light high-frequency sub-band; the output terminals of the low-frequency fusion module and the high-frequency fusion module are both connected to the input terminal of the inverse wavelet reconstruction module. The input and output terminals of the inverse wavelet reconstruction module, residual fusion module, and feature adaptation module are connected in sequence; the residual fusion module includes a learnable intensity parameter γ and a first weighted superposition module, and its other input terminal is connected to the visible light feature map. ; The inverse wavelet reconstruction module is used to perform inverse wavelet reconstruction on the low-frequency fusion result and the high-frequency fusion result to obtain the reconstructed fusion feature map; the residual fusion module is used to process the reconstructed fusion feature map using a learnable intensity parameter γ and then combine it with the visible light feature map. Weighted fusion is performed to obtain a fused residual feature map; the feature adaptation module is used to unify the distribution and smooth the features of the fused residual feature map to obtain multi-scale features. Step 5: Use the detection head unit to perform spatial decoupling and spatial attention enhancement on the first multi-scale feature, the second multi-scale feature and the third multi-scale feature respectively to obtain the final target detection result and complete the UAV detection.

6. The multimodal detection method for small-scale unmanned aerial vehicles according to claim 5, characterized in that, In step 4, the first DWBF module in the neck fusion unit is used to process the visible light feature map. and infrared light feature map The specific steps for frequency domain decoupling and fusion are as follows: S1. Use the first depthwise convolutional layer in the pre-alignment module of the first DWBF module to process the visible light feature map. Channel alignment is performed, and the signal is then input into the first wavelet decomposition module for decomposition to obtain the visible light low-frequency subband. and visible light high-frequency subband , , ; The infrared feature map is processed using the second deep convolutional layer in the pre-aligned module. Channel alignment is performed, and the signal is then input into the second wavelet decomposition module for decomposition to obtain the infrared low-frequency subband. Infrared high-frequency subband , , ; S2. Using a low-frequency fusion module to optimize the low-frequency subband of visible light. and infrared low-frequency subband The fusion process yields low-frequency fusion results. ; S2.1 Using spatially weighted generation units for visible light low-frequency subband and infrared low-frequency subband The weights α and β are obtained through processing. S2.2, Use the first weighted channel in the weighted fusion unit to perform fusion on the visible light low-frequency subband. The result is obtained by fusing the result with weight α. Using the second weighting channel for the infrared low-frequency subband The result is then fused with the weight β to obtain the second fusion result. The first fusion result and the second fusion result are fused using a low-frequency fusion unit to obtain the low-frequency fusion result. ; S3. Using a high-frequency fusion module to optimize visible light high-frequency subbands. , , and infrared high-frequency subband , , The fusion process is performed to obtain high-frequency fusion results. , and ; S3.1 Using physical rule fusion units for visible light high-frequency subband , , and infrared high-frequency subband , , Perform maximum absolute value selection fusion to obtain the selection fusion result. , and ; S3.2, Use the feature stitching module to select the fusion results. , and After fusion, the input is processed by the channel attention module to generate a channel attention map; S3.

3. Use the element-wise multiplication module to fuse the channel attention map with the selection result. , , The high-frequency feature map is obtained by fusing the features. ; S3.

4. Use the high-frequency subband splitting module Split to split the fused high-frequency feature map. Decomposed into high-frequency fusion results , and ; S4. Use the inverse wavelet reconstruction module to fuse the low-frequency results. High-frequency fusion results , and Perform inverse wavelet reconstruction to obtain the fused feature map. ; S5. Use the learnable intensity parameter γ in the residual fusion module to fuse the feature map. The data is processed and then compared with the visible light feature map. The samples are input into the first weighted overlay module for fusion to obtain the fused residual feature map; S6. Use the feature adaptation module to perform distribution unification and feature smoothing on the fused residual feature map to obtain the first multi-scale feature.

7. A multimodal detection method for small-scale unmanned aerial vehicles according to claim 6, characterized in that, Step 5 specifically involves: Step 5.1: Use the first upsampling module in the detection head unit to upsample the third multi-scale feature to obtain feature map U1; use the first feature stitching module to stitch feature map U1 and the second multi-scale feature to obtain feature map C1; use the fifth C3k2 module to process feature map C1 to obtain feature map F1. Step 5.2: Upsample feature map F1 using the second upsampling module to obtain feature map U2; then concatenate feature map U2 and the first multi-scale feature using the second feature concatenation module to obtain feature map C2. The feature map C2 is processed using the sixth C3k2 module to obtain the feature map F2; Step 5.3: Use the first ASCA module to perform spatial attention enhancement on feature map F2 to obtain feature map A2; output feature map A2 through the first detection module; Step 5.4: Use the first downsampling module to downsample feature map A2 to obtain feature map D1; use the third feature stitching module to stitch feature map D1 and feature map F1 along the channel dimension to obtain feature map C3; The seventh module C3k2 is used to process C3 to obtain feature map F3; Step 5.5: Use the second ASCA module to perform spatial attention enhancement on feature map F3 to obtain feature map A1; output feature map A1 through the second detection module; Step 5.6: Use the second downsampling module to downsample feature map A1 to obtain feature map D2; use the fourth feature stitching module to stitch D2 and the third multi-scale feature along the channel dimension to obtain feature map C4; use the eighth C3k2 module to process C4 to obtain feature map F4; output feature map F4 through the third detection module. The first ASCA module and the second ASCA module both include a global context modeling unit, a horizontal strip convolution unit, a vertical strip convolution unit, a feature fusion unit, an activation map generation unit, and an attention weighting unit; the input ends of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all used to connect to the input feature map. The global context modeling unit is used to perform global context expansion on the input feature map to obtain an expanded global context feature map; the horizontal strip convolution unit is used to extract long-distance dependency features in the horizontal direction to obtain a horizontal feature map; the vertical strip convolution unit is used to extract long-distance dependency features in the vertical direction to obtain a vertical feature map. The outputs of the global context modeling unit, the horizontal strip convolution unit, and the vertical strip convolution unit are all connected to the input of the feature fusion unit. The input and output terminals of the feature fusion unit, activation map generation unit, and attention weighting unit are connected in sequence; the other input terminal of the attention weighting unit is used to connect to the input feature map; the feature fusion unit is used to perform element-wise addition operations on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map; the activation map generation unit is used to process the fused feature map to generate a spatial attention map; the attention weighting unit is used to perform element-wise multiplication operations on the input feature map and the spatial attention map; The global context modeling unit includes an adaptive average pooling layer, a fourth 1×1 convolutional layer, and a third batch normalization layer, with the input and output connected in sequence; the horizontal strip convolutional unit and the vertical strip convolutional unit each include a 1×11 convolutional layer. The feature maps A1, A2, and F4 are the final detection results.

8. A multimodal detection method for small-scale unmanned aerial vehicles according to claim 7, characterized in that, Step 5.3 specifically involves: Step 5.3.1: Input feature map F2 into the global context modeling unit of the first ASCA module to obtain the global context feature map; expand the global context feature map through a broadcast mechanism to obtain the expanded global context feature map; Step 5.3.2: Input feature map F2 into the horizontal strip convolutional unit of the first ASCA module to extract long-distance dependency features in the horizontal direction and obtain the horizontal feature map; Step 5.3.3: Input feature map F2 into the vertical strip convolutional unit of the first ASCA module to extract long-range dependent features in the vertical direction and obtain the vertical feature map. Step 5.3.4: Use the feature fusion unit to perform element-wise addition on the horizontal feature map, vertical feature map, and expanded global context feature map to generate a fused feature map; Step 5.3.5: Use the activation mapping generation unit to process the fused feature map into a spatial attention map; Step 5.3.6: Perform element-wise multiplication on feature map F2 and spatial attention map using the attention weighting unit to generate feature map A2; output feature map A2 through the first detection module; The size of the global context feature map is (B, C, 1, 1); the sizes of the expanded global context feature map, horizontal feature map, vertical feature map, fused feature map, and spatial attention map are all (B, C, H, W).

Images