A cross-modal unmanned aerial vehicle detection method based on an environment driving strategy

Abstract

This invention discloses a cross-modal UAV detection method based on an environment-driven strategy, belonging to the field of UAV detection technology, and includes the following steps: preparing a dataset; designing a feature fusion module (MFFB) based on a state-space model; constructing an MDUAV-YOLO network model; training the network model; designing a composite loss function and evaluation index; and fine-tuning the model. The technical solution of this invention designs a cross-modal dual-branch MDUAV-YOLO target detection network, achieving adaptive adjustment of modal contribution through a dynamic gating mechanism; designs a state-space feature fusion module (MFFB), utilizing a state-space model to achieve deep interaction and complementarity of visible light and infrared features in complex dimensions; and designs a composite loss function specifically adapted for UAV detection. Experimental data demonstrates good performance in UAV target detection tasks under various complex backgrounds.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a cross-modal UAV detection method based on an environment-driven strategy. Background Technology

[0002] With the rapid development of target detection algorithms, numerous methods for detecting drones have emerged. However, two main problems remain. Firstly, most existing drone detection algorithms are single-modal, which significantly limits their performance in complex environments: drones account for a small percentage of pixels in infrared images, making edge contour feature extraction difficult; and the accuracy of visible light image detection drops sharply due to complex environments such as nighttime, sandstorms, and haze. Occlusion by clouds, birds, and other objects also reduces recall. Secondly, existing multimodal fusion methods are mostly based on convolutional neural networks (CNNs) or Transformer architectures. CNNs have lower computational overhead but lack a global receptive field, while Transformer networks have a global receptive field but higher computational complexity. State-space models can capture long-range dependencies with linear complexity. Maintaining low computational overhead while improving multimodal fusion efficiency and small target detection feature extraction efficiency using state-space models remains a challenge.

[0003] Chinese patent publication number "CN119716839A" is titled "A High-Precision Detection Method for Anti-UAVs Based on Multimodal Data Fusion using CNYOLOv5". This method uses the YOLOv5 network as its foundation, introducing a Concentration Attention (CBAM) module into the backbone network. CBAM provides the model with a global receptive field, adaptively identifying key regions in the input image or feature map while suppressing background interference. A Neural Architecture Search-Optimized Feature Pyramid Network (NASFPN) is introduced to replace the FPN module in the original YOLOv5 model's Neck section for feature fusion, enabling the matching and fusion of features extracted from different modalities through different model levels. While this method emphasizes the global receptive field's contribution to feature extraction, it doesn't adequately address the need for local detail feature extraction for small, indistinct UAVs at medium to long distances. Furthermore, extracting features from different modalities at different model levels leads to significant information loss, and the method fails to consider the adaptability of multimodal feature information to different environments to better improve fusion and detection performance. Summary of the Invention

[0004] The technical solution of this invention to solve the above-mentioned technical problems is to provide a cross-modal UAV detection method based on an environment-driven strategy, comprising the following steps:

[0005] S1. Prepare datasets: Obtain two cross-modal drone datasets and preprocess them. Cross-modal drone dataset one is used for network training, and cross-modal drone dataset two is used for model fine-tuning and testing.

[0006] S2. Design a feature fusion module MFFB based on a state-space model: The module includes the state-space operator SS2D, depthwise convolution, normalization layer, linear layer and activation function;

[0007] S3. Construct the MDUAVYOLO network model: Build a multimodal detection framework including a dual-branch backbone network, a cross-modal feature fusion network, a neck network, and a head network;

[0008] S4. Training the network model: Train the network model using the cross-modal drone dataset;

[0009] S5. Design composite loss function and evaluation metrics: The composite loss function includes a category classification loss using BCE loss and a bounding box regression loss using DFL loss plus SIoU loss; the evaluation metrics include precision P, recall R, mAP50, mAP, frame rate FPS, number of parameters Params, and computational complexity GFLOPs.

[0010] S6. Fine-tuning the model: Two pairs of models are fine-tuned using a cross-modal UAV dataset, the fine-tuned model is saved and target detection is performed;

[0011] S7. Solidify the network parameters and save the final target detection model.

[0012] Furthermore, in S1, the first cross-modal UAV dataset is the AntiUAV dataset, and the second cross-modal UAV dataset is the self-made multimodal UAV dataset;

[0013] Preprocessing includes registering unaligned images and augmenting the dataset through inversion, translation, and symmetry operations.

[0014] Furthermore, in S2, the MFFB module is divided into an infrared sub-module and a visible light sub-module; the infrared feature map is subjected to big kernel sparsification operation and SS2D operator to extract features, and then is initially fused with the features transmitted by the visible light sub-module, and then is linearly projected and fused with the projected features of the visible light sub-module to output the final multimodal features; the visible light sub-module performs symmetric operation.

[0015] Furthermore, the MFFB module employs a dynamic gating mechanism, implementing the environment-driven strategy through the following steps:

[0016] (a) Project the infrared feature map and the visible light feature map onto the high-dimensional state space respectively:

[0017]

[0018] Where f(.) represents projecting the features onto a complex state space; and Represents the characteristics of complex states;

[0019] (b) Projection and Obtain gating parameters and

[0020]

[0021] Where α and β represent the adaptive adjustment operation of the image for features in a multimodal task, respectively;

[0022] (c) Modulation feature interaction:

[0023]

[0024] in, and These represent the state features of the visible light and infrared images after the initial feature interaction, respectively; · represents the element-wise product.

[0025] (d) Cross-modal deep fusion: The bimodal features are respectively elevated to a high-dimensional complex state space for deep fusion. Adaptive parameters are used during cross-branch information interaction to enable full information complementarity between visible light and infrared images, thereby obtaining complementary features. and

[0026]

[0027] Among them, f out (.) denotes a simple linear projection, μ1 = 0.64, μ2 = 0.49.

[0028] Furthermore, the workflow of the dual-branch backbone network in S3 includes:

[0029] The visible light branch and the infrared branch each extract basic features through a three-level local convolutional layer;

[0030] The VSS module is used to globally enhance the basic features;

[0031] The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features are used to generate the next level feature map through residual connections.

[0032] Hierarchical feature maps are processed in two ways: one way is weighted and fused across branches and then input into the neck network, and the other way is passed to the next level through the local feature enhancement unit;

[0033] The process is iterated twice with dual-path processing. The deep features are then output to the neck network and cross-modal fusion module after spatial pyramid pooling.

[0034] Furthermore, the local convolutional layer adopts the C3K2 module, which includes depthwise separable convolution and channel attention mechanism; the VSS module achieves multi-scale feature stabilization through parallel dilated convolutional layers and global average pooling layers.

[0035] Furthermore, the neck network adopts a bidirectional feature pyramid structure, which aggregates multi-scale features through a dual-path fusion method from top to bottom and bottom to top, and embeds a coordinate attention mechanism in the fusion layer.

[0036] Compared with existing technologies, this invention provides a cross-modal UAV detection method based on an environment-driven strategy, which has the following advantages:

[0037] 1. This invention proposes a novel cross-modal target detection network architecture, MDUAV-YOLO, with a backbone network designed as a dual-branch structure for visible light and infrared. By combining the local detail information extraction capability of convolution with the global information extraction capability of the state space model, the efficiency of UAV detection is improved.

[0038] 2. This invention proposes a state-space feature fusion module MFFB, which effectively alleviates the spatial mismatch problem of cross-modal features by using the high-dimensional state evolution mechanism of the state-space model, and significantly improves the deep fusion capability of multimodal features.

[0039] 3. This invention improves the adaptability to different complex environments by designing an environment-driven strategy and utilizing a dynamic gating mechanism to achieve adaptive adjustment of modal contribution.

[0040] 4. This invention designs a novel composite loss function to specifically address issues such as the small size of UAV targets and unclear boundaries in complex environments, which can easily lead to confusion with the background. It utilizes more suitable classification loss and bounding box regression loss. The SIoU loss, composed of angle loss, distance loss, shape loss, and IoU loss, is more suitable for small target detection tasks in complex backgrounds. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0042] Figure 1This is a flowchart of the steps of the cross-modal UAV detection method based on environment-driven strategy of the present invention;

[0043] Figure 2 This is a schematic diagram of the MDUAV-YOLO network architecture of the present invention;

[0044] Figure 3 This is a schematic diagram of the MFFB structure designed for this invention;

[0045] Figure 4 This is a schematic diagram of the VSS global feature extraction module structure of the present invention;

[0046] Figure 5 This is a schematic diagram of the neck network structure of the present invention;

[0047] Figure 6 This is a schematic diagram of the C3K2 local feature extraction module structure of the present invention;

[0048] Figure 7 This is a schematic diagram showing a qualitative comparison between the cross-modal UAV detection method based on the environment-driven strategy of the present invention and existing technologies. Detailed Implementation

[0049] This invention proposes a cross-modal UAV detection method based on an environment-driven strategy, aiming to solve the problems of insufficient single-modal adaptability, poor multimodal feature fusion effect, and low detection accuracy of low, slow, and small targets in complex environments.

[0050] The cross-modal UAV detection method based on an environment-driven strategy proposed in this invention will be described below in specific embodiments:

[0051] Example 1:

[0052] A cross-modal UAV detection method based on an environment-driven strategy, such as Figure 1 As shown, it includes the following steps:

[0053] S1. Prepare datasets: Obtain two cross-modal drone datasets and preprocess them. Cross-modal drone dataset one is used for network training, and cross-modal drone dataset two is used for model fine-tuning and testing.

[0054] S2. Design a feature fusion module MFFB based on a state-space model: The module includes the state-space operator SS2D, depthwise convolution, normalization layer, linear layer and activation function;

[0055] S3. Construct the MDUAVYOLO network model: Build a multimodal detection framework including a dual-branch backbone network, a cross-modal feature fusion network, a neck network, and a head network;

[0056] S4. Training the network model: Train the network model using the cross-modal drone dataset;

[0057] S5. Design composite loss function and evaluation metrics: The composite loss function includes a category classification loss using BCE loss and a bounding box regression loss using DFL loss plus SIoU loss; the evaluation metrics include precision P, recall R, mAP50, mAP, frame rate FPS, number of parameters Params, and computational complexity GFLOPs.

[0058] S6. Fine-tuning the model: Two pairs of models are fine-tuned using a cross-modal UAV dataset, the fine-tuned model is saved and target detection is performed;

[0059] S7. Solidify the network parameters and save the final target detection model.

[0060] Furthermore, in S1, the first cross-modal UAV dataset is the AntiUAV dataset, and the second cross-modal UAV dataset is the self-made multimodal UAV dataset;

[0061] Preprocessing includes registering unaligned images and augmenting the dataset through inversion, translation, and symmetry operations.

[0062] Furthermore, in S2, the MFFB module is divided into an infrared sub-module and a visible light sub-module; the infrared feature map is subjected to big kernel sparsification operation and SS2D operator to extract features, and then is initially fused with the features transmitted by the visible light sub-module, and then is linearly projected and fused with the projected features of the visible light sub-module to output the final multimodal features; the visible light sub-module performs symmetric operation.

[0063] Furthermore, the MFFB module employs a dynamic gating mechanism, implementing the environment-driven strategy through the following steps:

[0064] (a) Project the infrared feature map and the visible light feature map onto the high-dimensional state space respectively:

[0065]

[0066] Where f(.) represents projecting the features onto a complex state space; and Represents the characteristics of complex states;

[0067] (b) Projection and Obtain gating parameters and

[0068]

[0069] Where α and β represent the adaptive adjustment operation of the image for features in a multimodal task, respectively;

[0070] (c) Modulation feature interaction:

[0071]

[0072] in, and These represent the state features of the visible light and infrared images after the initial feature interaction, respectively; · represents the element-wise product.

[0073] (d) Cross-modal deep fusion: The bimodal features are respectively elevated to a high-dimensional complex state space for deep fusion. Adaptive parameters are used during cross-branch information interaction to enable full information complementarity between visible light and infrared images, thereby obtaining complementary features. and

[0074]

[0075] Among them, f out (.) denotes a simple linear projection, μ1 = 0.64, μ2 = 0.49.

[0076] Furthermore, the workflow of the dual-branch backbone network in S3 includes:

[0077] The visible light branch and the infrared branch each extract basic features through a three-level local convolutional layer;

[0078] The VSS module is used to globally enhance the basic features;

[0079] The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features are used to generate the next level feature map through residual connections.

[0080] Hierarchical feature maps are processed in two ways: one way is weighted and fused across branches and then input into the neck network, and the other way is passed to the next level through the local feature enhancement unit;

[0081] The process is iterated twice with dual-path processing. The deep features are then output to the neck network and cross-modal fusion module after spatial pyramid pooling.

[0082] Furthermore, the local convolutional layer adopts the C3K2 module, which includes depthwise separable convolution and channel attention mechanism; the VSS module achieves multi-scale feature stabilization through parallel dilated convolutional layers and global average pooling layers.

[0083] Furthermore, the neck network adopts a bidirectional feature pyramid structure, which aggregates multi-scale features through a dual-path fusion method from top to bottom and bottom to top, and embeds a coordinate attention mechanism in the fusion layer.

[0084] Example 2:

[0085] A cross-modal UAV detection method based on an environment-driven strategy includes the following steps:

[0086] S1, Prepare the datasets: Prepare two cross-modal UAV datasets and preprocess them. Cross-modal UAV dataset one is used for network training and model fine-tuning, and cross-modal UAV dataset two is used for model testing;

[0087] The S1 cross-modal UAV dataset is an aerial background infrared image UAV detection and tracking dataset (Anti-UAV). This dataset was acquired simultaneously using dual spectra, with each pair containing visible light and infrared spectral video streams, covering six types of UAV targets from major manufacturers such as DJI and Parnot. It also includes daytime natural lighting and nighttime low-light environments, containing typical airspace background elements such as buildings, clouds, and vegetation. The total size reaches 318 high-resolution video pairs, generating 331,919 annotated frames. Each target corresponds to a label position in the image, and each image sequence corresponds to a label file.

[0088] The second multimodal drone dataset in S1 is a self-made multimodal drone dataset for complex scenarios. This dataset includes a small domestically produced drone and a medium-sized DJI drone. The dataset contains 17,341 images and 17,996 targets. Random rotation, translation, and symmetry geometric transformations were performed on the images, and brightness and contrast were adjusted to expand the dataset and improve its robustness and generalization ability in complex environments.

[0089] S2, Design the feature fusion module MFFB based on the state-space model: The structure of the feature fusion module MFFB is as follows: Figure 3 As shown, the symmetrical visible light feature map is composed of a normalization layer, a linear layer, an activation function, a depthwise dilated convolution, a depthwise convolution, a channel convolution, and an SS2D operator. The visible light feature map and the infrared feature map undergo normalization, a linear layer, and an activation function, followed by a large kernel sparsification operation. They are then fed into SS2D for cross-selective scanning. The resulting high-dimensional features are initially fused with the multimodal low-dimensional features of the branch. After fusion, they are projected into a linear layer and then subjected to a second deep fusion with the high-dimensional features of another modality. Simultaneously, a skip connection is made with the initial input feature map to output the final fused feature.

[0090] The MFFB module in S2 utilizes a gating mechanism to transform and fuse bimodal features. Specifically, the VSS module is first used to perform high-dimensional projection on different modal features. The VSS global feature extraction module, as shown... Figure 4 As shown below:

[0091]

[0092] Where f(.) represents projecting the features onto the complex state space. and It represents the characteristics of complex states.

[0093] They also projected it. and To obtain gating parameters and

[0094]

[0095] Where α and β represent the image adaptive adjustment operations for features in a multimodal task. Then, the equation... and The gating output is used for modulation and

[0096]

[0097] in, and These represent the state features of the visible light and infrared images after the initial feature interaction, respectively. · represents element-wise multiplication.

[0098] Finally, the bimodal features were respectively elevated to a high-dimensional complex state space for deep fusion. Furthermore, adaptive parameters were used during cross-branch information interaction to enable sufficient information complementarity between visible light and infrared images, thereby obtaining complementary features. and

[0099]

[0100] Where f out (.) represents a simple linear projection, and the results from experiments show that μ1 = 0.64 and μ2 = 0.49.

[0101] S3, Construct the MDUAV-YOLO network model: The overall structure of the model is based on the YOLOv11 network framework, as follows: Figure 2 As shown, it mainly includes three stages: single-modal feature extraction, dual-modal feature fusion, and neck and head detection. The backbone network consists of a dual-branch feature extraction backbone network and three state-space feature fusion blocks (MFFB). The neck network and detection head are used for cross-modal target detection. The neck network is shown below. Figure 5 As shown. The feature extraction network first extracts local features from visible light and infrared images using a convolutional feature extraction module (C3K2). The C3K2 local feature extraction module is as follows: Figure 6 As shown, they are respectively represented as and

[0102]

[0103] in Let H, W, and C represent the feature maps of the i-th layer of the visible light and infrared images, respectively, where H, W, and C represent the height, width, and number of channels of the feature map, respectively. RGB I IR These represent the input visible light and infrared images, respectively. ψ backbone Let ψ represent the feature extraction function. backbone , ψ backbone This represents the parameters used in the feature extraction process.

[0104] By inputting these two features into the MFFB (Multimodal Fusion Functional Model), multimodal features are fused in a complex state space. This reduces the differences between cross-modal features and enhances the representational consistency of the fused features. Specifically, these two local features undergo initial feature fusion in the initial module of the state-space fusion MFFB model. These interactive features are then fed into subsequent modules for deep feature fusion in the complex state space, generating corresponding complementary features. and By using the original features and Added separately to complementary feature formation and This is used to enhance local features. Then, the enhanced features are directly applied... and The fusion features are generated by addition. In the last three stages, MFFB is added to generate fusion features P3, P4 and P5. These features are used as inputs to the neck and head of YOLOv11 to generate the final detection results.

[0105] S5. Design the loss function and determine the evaluation metric: Improve the composite loss function, iterate and optimize the model multiple times until the values ​​of the category classification loss and bounding box regression loss reach the expected threshold, or the number of iterations reaches the preset threshold, the model ends training and the model parameters are saved; input the test images in the dataset into the network for testing, select the optimal evaluation metric to measure the test results, and evaluate the accuracy and performance of the model.

[0106] The loss function in S5 uses a composite loss function, including class classification loss and bounding box regression loss;

[0107] A multi-class discrimination mechanism is constructed using binary cross-entropy loss (BCE). The input feature map is divided into gridded regions to generate N×M dimensional anchor boxes. Multi-channel one-hot encoding is performed on each anchor box (encoding dimension K corresponds to the total number of target classes). The confidence probability distribution matrix of each channel is output through a fully connected layer, and a maximum value selection strategy is implemented. BCELoss can be expressed by the following formula:

[0108]

[0109] Where N is the total number of samples, L i Let y be the loss for the i-th sample. i p is the true label of the i-th sample. i The model predicts that sample i belongs to the positive class (y). i The probability of (=1).

[0110] This invention employs a collaborative optimization strategy combining the Dynamic Focusing (DFL) mechanism (Distribution Focal Loss) and the Spatial Intersection over Union (SIoU) multi-dimensional spatial constraint. The DFL loss function establishes a discretized coordinate probability distribution model, and optimizes the neighborhood probability of the target location through cross-entropy loss. The DFL loss can be expressed by the following formula:

[0111] L DFL (S i ,S i+1 )=-[(y i+1 -y)log(S i )+(yy i )log(S i+1 )];

[0112] Where S i y represents the model's predicted probability for adjacent discretized intervals, and y represents the ground truth value of the bounding box in object detection. i These are the adjacent reference points of the discretized interval.

[0113] The Spatial Intersection over Union (SIoU) loss significantly optimizes bounding box regression accuracy by introducing orientation-aware constraints and coupling with multi-dimensional spatial features. An azimuth cost loss function based on vector angle is established, which solves the problem of traditional methods failing to detect at 45° by calculating the directional projection of the line connecting the centers of the predicted and ground truth bounding boxes. By introducing a diagonal standard deviation weighting factor and a nonlinear adjustment parameter, the detection accuracy after occlusion is significantly improved, making it particularly suitable for small target detection scenarios such as UAVs. Specifically, SIoU loss consists of angle loss, distance loss, shape loss, and IoU loss. SIoU loss can be expressed by the following formula:

[0114]

[0115] Where Δ is the distance loss, Λ is the angle loss, Ω is the shape loss, and c h ν is the vertical distance between the center points of the predicted bounding box and the ground truth bounding box, σ is the standard deviation of the diagonal lengths of the predicted bounding box and the ground truth bounding box, ν is a hyperparameter that adjusts the decay rate, and θ is a hyperparameter that adjusts the degree of nonlinearity.

[0116] The total loss of this paper can be expressed as:

[0117] L=λ1L BCE +λ2L DFL +λ3L SIoU ;

[0118] In the experiment, the detection effect was best when λ1, λ2, and λ3 were 0.5, 1, and 5.8, respectively.

[0119] The evaluation metrics in S5 use P (precision), R (recall), mAP50 (average precision with an IoU threshold of 0.5), mAP, FPS (frame rate), number of parameters, and GFLOPs (model computational complexity) as core metrics.

[0120]

[0121] TP represents a true positive, which is the intersection-union ratio (IoU) of the detector's predicted bounding box and the true bounding box greater than the IoU threshold; otherwise, it is considered a false positive (FP). A false negative (FN) means that a real target exists but is not detected. Equation (18) shows that AP is the integral of the precision-recall curve for each class. mAP50 is the average AP value for all classes when IoU = 0.50, calculated in Equation (17). Similarly, mAP95 is the average AP value for all classes when IoU = 0.95. mAP is the primary challenge metric, which can be expressed as the average when IoU = 0.50:0.05:0.95.

[0122] S6, Fine-tuning the model: Retrain the model using the second infrared dataset and fine-tune the hyperparameters to further improve the performance of the detection network.

[0123] S7, Save Model: After all training is completed, fix all parameters and determine the multimodal object detection model.

[0124] This invention proposes a novel cross-modal target detection network architecture, MDUAV-YOLO, with a backbone network designed as a dual-branch structure for visible light and infrared. By combining the local detail information extraction capability of convolution with the global information extraction capability of the state space model, it improves the detection efficiency of UAVs.

[0125] This invention proposes a state-space feature fusion module (MFFB), which effectively alleviates the spatial mismatch problem of cross-modal features by utilizing the high-dimensional state evolution mechanism of the state-space model, and significantly improves the deep fusion capability of multimodal features.

[0126] This invention improves the adaptability to different complex environments by designing an environment-driven strategy and utilizing a dynamic gating mechanism to achieve adaptive adjustment of modal contribution.

[0127] This invention designs a novel composite loss function to specifically address issues such as small drone targets and unclear boundaries that easily lead to confusion with the background in complex environments. It employs more suitable classification and bounding box regression losses. The SIoU loss, composed of angle loss, distance loss, shape loss, and IoU loss, is better suited for small target detection tasks in complex backgrounds.

[0128] To verify the superiority of the method of this invention, the method proposed in this invention was compared with existing advanced methods on a self-made dataset for drone detection. The detection results for the two scenarios are as follows: Figure 7 As shown. Existing method 1 uses the CFT (Cross Modality Fusion Transformer) network to capture long-range dependencies and integrate global contextual information through the Transformer architecture. Its core innovation lies in its ability to simultaneously and naturally perform intra-modal and inter-modal feature fusion through a self-attention mechanism. This method can effectively extract interaction information between visible light and infrared modalities, improving the performance of multispectral target detection. Existing method 2 proposes a feature fusion framework with a dual cross-attention mechanism using the ICAFusion network. This framework uses a query-guided cross-attention mechanism to enhance the discriminative power of target features. An iterative interaction mechanism is introduced to overcome the high parameters and computational cost caused by stacking Transformer blocks, achieving parameter sharing between multimodal Transformers at the block level. This method is highly versatile, effective, and fast inference, and is suitable for various practical scenarios. The cross-modal UAV detection method based on an environment-driven strategy proposed in this invention combines local detail information and global information to enhance feature extraction capabilities and enhance feature fusion capabilities under different modalities in high-dimensional space. Figure 7 A comparison of the detection results shows that it achieved the best detection performance in small target detection tasks in real and complex scenarios.

[0129] Comparative experiments conducted on the same dataset show that, compared with the two existing methods and the single-modal detection method, the proposed method in this invention demonstrates improvements in relevant evaluation metrics (such as mAP, mAP50, and model computational complexity) in cross-modal UAV target detection tasks. Specific comparative data are shown in Table 1. During the testing phase, the detection method of this invention achieved a 5.6% improvement in accuracy compared to the suboptimal method, while significantly reducing computational cost. These results validate the feasibility of the proposed method and its performance advantages over existing technologies.

[0130] Table 1. Comparison of data between this application and prior art:

[0131]

[0132] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal UAV detection method based on an environment-driven strategy, characterized in that, Includes the following steps: S1. Prepare datasets: Obtain two cross-modal drone datasets and preprocess them. Cross-modal drone dataset one is used for network training, and cross-modal drone dataset two is used for model fine-tuning and testing. S2. Design a feature fusion module MFFB based on a state-space model: The module includes the state-space operator SS2D, depthwise convolution, normalization layer, linear layer and activation function; S3. Construct the MDUAV-YOLO network model: Build a multimodal detection framework including a dual-branch backbone network, a cross-modal feature fusion network, a neck network, and a head network; S4. Training the network model: Train the network model using the cross-modal drone dataset; S5. Design composite loss function and evaluation metrics: The composite loss function includes a category classification loss using BCE loss and a bounding box regression loss using DFL loss plus SIoU loss; The evaluation metrics include precision P, recall R, mAP50, mAP, frame rate FPS, number of parameters Params, and computational complexity GFLOPs. S6. Fine-tuning the model: Two pairs of models are fine-tuned using a cross-modal UAV dataset, the fine-tuned model is saved and target detection is performed; The MFFB module employs a dynamic gating mechanism and implements the environment-driven strategy through the following steps: (a) Project the infrared feature map and the visible light feature map onto the high-dimensional state space respectively: ； in, This represents projecting features into a complex state space; and Represents characteristics of complex states; (b) Projection and Obtain gating parameters and : ； in, , These represent the adaptive adjustment operations of images based on features in multimodal tasks; (c) Modulation feature interaction: ； ； in, and These represent the state features of the visible light and infrared images after the initial feature interaction, respectively; · represents the element-wise product. (d) Cross-modal deep fusion: The bimodal features are respectively elevated to a high-dimensional complex state space for deep fusion. Adaptive parameters are used during cross-branch information interaction to enable full information complementarity between visible light and infrared images, thereby obtaining complementary features. and : ； in, This represents a simple linear projection. =0.64, =0.

49.

2. The cross-modal UAV detection method based on an environment-driven strategy according to claim 1, characterized in that: In S1, the first cross-modal UAV dataset is the AntiUAV dataset, and the second cross-modal UAV dataset is the self-made multimodal UAV dataset. Preprocessing includes registering unaligned images and augmenting the dataset through inversion, translation, and symmetry operations.

3. The cross-modal UAV detection method based on an environment-driven strategy according to claim 1, characterized in that: The MFFB module in S2 is divided into an infrared sub-module and a visible light sub-module. The infrared feature map is first fused with the features transmitted by the visible light sub-module after being subjected to large kernel sparsification operation and SS2D operator extraction. Then, after linear projection, it is fused with the projected features of the visible light sub-module to output the final multimodal features. The visible photonic module performs symmetric operations.

4. The cross-modal UAV detection method based on an environment-driven strategy according to claim 1, characterized in that: The workflow of the dual-branch backbone network in S3 includes: The visible light branch and the infrared branch each extract basic features through a three-level local convolutional layer; The VSS module is used to globally enhance the basic features; The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features are used to generate the next level feature map through residual connections. Hierarchical feature maps are processed in two ways: one way is weighted and fused across branches and then input into the neck network, and the other way is passed to the next level through the local feature enhancement unit; The process is iterated twice with dual-path processing. The deep features are then output to the neck network and cross-modal fusion module after spatial pyramid pooling.

5. The cross-modal UAV detection method based on an environment-driven strategy according to claim 4, characterized in that: The local convolutional layer uses the C3K2 module, which includes depthwise separable convolution and channel attention mechanisms; the VSS module achieves multi-scale feature stabilization through parallel dilated convolutional layers and global average pooling layers.

6. The cross-modal UAV detection method based on an environment-driven strategy according to claim 1, characterized in that: The neck network adopts a bidirectional feature pyramid structure, which aggregates multi-scale features through a dual-path fusion method of top-down and bottom-up, and embeds a coordinate attention mechanism in the fusion layer.

Images