Cross-modal unmanned aerial vehicle detection method based on environment driving strategy

Abstract

The invention discloses a cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy, and relates to the technical field of unmanned aerial vehicle detection, and the method comprises the following steps: preparing a data set; designing a feature fusion module MFFB based on a state space model; an MDUAV-YOLO network model is constructed; training a network model; designing a composite loss function and an evaluation index; finely adjusting the model. According to the technical scheme of the invention, a cross-modal double-branch MDUAV-YOLO target detection network is designed, and the adaptive adjustment of the modal contribution degree is realized through a dynamic gating mechanism; designing a state space feature fusion module MFFB, and realizing deep interaction and complementation of visible light and infrared features in a complex dimension by using a state space model; and designing a composite loss function to pertinently adapt to unmanned aerial vehicle detection. Experimental data prove that the method shows good performance in unmanned aerial vehicle target detection tasks under various complex backgrounds.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy. BACKGROUND

[0002] At present, with the vigorous development of target detection algorithms, there are numerous detection methods for unmanned aerial vehicles. However, there are the following two problems. On the one hand, most of the existing unmanned aerial vehicle detection algorithms are single-modal algorithms, and the detection performance of single-modal algorithms is significantly limited in complex environments: the pixel ratio of unmanned aerial vehicles in infrared images is small, making it difficult to extract edge contour features; the detection accuracy of visible light images decreases sharply with complex environments such as night, sand, and haze, and objects such as clouds and birds also cause the recall rate to decrease. On the other hand, the existing multi-modal fusion methods are mostly based on convolutional neural networks (CNN) or Transformer architecture, and the convolutional network has less computational overhead but lacks a global receptive field, and the Transformer network has a global receptive field but has high computational complexity. It is still difficult to use state space models to maintain low computational overhead while improving multi-modal fusion efficiency and small target detection feature extraction efficiency.

[0003] Chinese Patent Publication No. CN119716839A, entitled "A multi-modal data fusion anti-unmanned aerial vehicle high-precision detection method based on CNYOLOv5", introduces an attention mechanism module (CBAM) into the backbone network based on the YOLOv5 network, which can provide a global receptive field for the model, adaptively identify the key areas that play a role globally in the input image or feature map, and suppress the interference of background information; a neural architecture search optimized feature pyramid network (NASFPN) is introduced to replace the FPN module in the Neck part of the original YOLOv5 model for feature fusion, which can match and fuse the features extracted from different modal data by different levels of the model. This method focuses on the help of global receptive field for feature extraction, but does not fully consider the problem of local detail feature extraction for small and fuzzy unmanned aerial vehicles in the middle and long distance. This method extracts features from different modalities using different levels of the model, which will result in a large amount of information loss, and does not consider the adaptation of multi-modal feature information to different environments to better improve the fusion and detection effect. SUMMARY

[0004] The technical solution of the present application to solve the above technical problems is to provide a cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy, comprising the following steps:

[0005] S1. Preparing dataset: obtaining and preprocessing two kinds of cross-modal UAV datasets, wherein the cross-modal UAV dataset one is used for network training, and the cross-modal UAV dataset two is used for model fine-tuning and testing;

[0006] S2. Designing a feature fusion module MFFB based on a state space model: the module includes a state space operator SS2D, a deep convolution, a normalization layer, a linear layer and an activation function;

[0007] S3. Building an MDUAVYOLO network model: building a multi-modal detection framework including a double-branch backbone network, a cross-modal feature fusion network, a neck network and a head network;

[0008] S4. Training the network model: training the network model using the cross-modal UAV dataset one;

[0009] S5. Designing a composite loss function and evaluation index: the composite loss function includes a class classification loss using a BCE loss, and a bounding box regression loss using a DFL loss and a SIoU loss; the evaluation index includes precision P, recall R, mAP50, mAP, frame rate FPS, parameter quantity Params and calculation complexity GFLOPs;

[0010] S6. Fine-tuning the model: fine-tuning the model using the cross-modal UAV dataset two, saving the fine-tuned model and performing target detection;

[0011] S7. Solidifying network parameters, saving the final target detection model.

[0012] Further, in the S1, the cross-modal UAV dataset one is AntiUAV dataset, and the cross-modal UAV dataset two is a self-made multi-modal UAV dataset;

[0013] The preprocessing includes registering the unaligned images, and expanding the dataset through inversion, translation and symmetry operations.

[0014] Further, in the S2, the MFFB module is divided into an infrared submodule and a visible light submodule; the infrared feature map is sequentially subjected to a large kernel sparsification operation and an SS2D operator to extract features, then the features are initially fused with the features transmitted by the visible light submodule, and then after linear projection, the features are secondarily fused with the projected features of the visible light submodule, and finally the multi-modal features are output; the visible light submodule performs a symmetry operation.

[0015] Further, the MFFB module adopts a dynamic gating mechanism to realize an environment-driven strategy through the following steps:

[0016] (a) Projecting the infrared feature map and the visible light feature map into a high-dimensional state space, respectively:

[0017]

[0018] where f(.) denotes the projection of the feature into the complex state space; and denote the complex state feature;

[0019] (b) projection and obtain the gating parameter and

[0020]

[0021] where α, β respectively represent the image adaptive adjustment operation in the multi-modal task;

[0022] (c) modulation of feature interaction:

[0023]

[0024] where and denote the state features of the visible light and infrared images after the initial interaction of the features; • is the element-wise product;

[0025] (d) cross-modal deep fusion: the bi-modal features are respectively pulled up to a high-dimensional complex state space for deep fusion, and adaptive parameters are used in the cross-branch information interaction process to make the visible light and infrared images fully complementary in information, obtaining complementary features and

[0026]

[0027] where f out (.) denotes a simple linear projection, μ1=0.64, μ2=0.49.

[0028] Further, the workflow of the double-branch backbone network in S3 includes:

[0029] The visible light branch and the infrared branch respectively extract basic features through three-level local convolutional layers;

[0030] The VSS module is used to globally enhance the basic features;

[0031] The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features generate the next level feature map through residual connection;

[0032] The level feature map is processed in two ways: one way is to input the neck network after cross-branch weighted fusion, and the other way is to pass through the local feature enhancement unit to the next level;

[0033] The two-way processing is iteratively performed twice, and the deep features are output to the neck network and the cross-modal fusion module after being spatially pyramid-pooled.

[0034] Further, the local convolutional layer adopts a C3K2 module, including a depth separable convolution and a channel attention mechanism; and the VSS module realizes multi-scale feature stabilization through a parallel hollow convolutional layer and a global average pooling layer.

[0035] Further, the neck network adopts a bidirectional feature pyramid structure, aggregates multi-scale features through a top-down and bottom-up two-way fusion mode, and embeds a coordinate attention mechanism in the fusion layer.

[0036] Compared with the prior art, the present application provides a cross-modal UAV detection method based on an environment-driven strategy, which has the following beneficial effects:

[0037] 1. The present application proposes a new cross-modal target detection network architecture MDUAV-YOLO, the backbone network is designed as a dual-branch structure of visible light and infrared, which combines the local detail information extraction ability of convolution and the global information extraction ability of state space model, and improves the UAV detection efficiency.

[0038] 2. The present application proposes a state space feature fusion module MFFB, which effectively alleviates the spatial mismatch problem of cross-modal features through the high-dimensional state evolution mechanism of the state space model, and significantly improves the deep fusion ability of multi-modal features.

[0039] 3. The present application designs an environment-driven strategy, and uses a dynamic gating mechanism to realize adaptive adjustment of the modal contribution degree, and improves the adaptability to different complex environments.

[0040] 4. The present application designs a new composite loss function, which uses more suitable classification loss and bounding box regression loss to solve the problems of small UAV target, unclear boundary and easy confusion with background in complex environment. The SIoU loss is composed of angle loss, distance loss, shape loss and IoU loss, which is more suitable for small target detection task in complex background. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without creative labor.

[0042] Figure 1A step flow chart of the cross-modal unmanned aerial vehicle detection method based on the environment-driven strategy of the present application;

[0043] Figure 2 A schematic diagram of the MDUAV-YOLO network architecture of the present application;

[0044] Figure 3 A schematic diagram of the MFFB structure designed by the present application;

[0045] Figure 4 A schematic diagram of the VSS global feature extraction module structure of the present application;

[0046] Figure 5 A schematic diagram of the neck network structure of the present application;

[0047] Figure 6 A schematic diagram of the C3K2 local feature extraction module structure of the present application;

[0048] Figure 7 A qualitative comparison diagram between the cross-modal unmanned aerial vehicle detection method based on the environment-driven strategy of the present application and the prior art scene. DETAILED DESCRIPTION

[0049] The present application proposes a cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy, aiming to solve the problems of insufficient single-modal adaptability, poor multi-modal feature fusion effect, and low detection precision of small, slow and small targets in complex environments.

[0050] The cross-modal unmanned aerial vehicle detection method based on the environment-driven strategy proposed by the present application will be described in specific embodiments as follows:

[0051] Embodiment 1:

[0052] A cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy, as shown in Figure 1 , comprising the following steps:

[0053] S1. Prepare the data set: obtain two cross-modal unmanned aerial vehicle data sets and preprocess them, wherein the cross-modal unmanned aerial vehicle data set one is used for network training, and the cross-modal unmanned aerial vehicle data set two is used for model fine-tuning and testing;

[0054] S2. Design a feature fusion module MFFB based on a state space model: the module includes a state space operator SS2D, a deep convolution, a normalization layer, a linear layer and an activation function;

[0055] S3. Build an MDUAVYOLO network model: build a multi-modal detection framework including a double-branch backbone network, a cross-modal feature fusion network, a neck network and a head network;

[0056] S4. Training the network model: the network model is trained using the cross-modal UAV dataset one;

[0057] S5. Designing a composite loss function and evaluation index: the composite loss function includes a class classification loss using a BCE loss and a bounding box regression loss using a DFL loss and a SIoU loss; the evaluation index includes precision P, recall R, mAP50, mAP, frame rate FPS, parameter quantity Params, and calculation complexity GFLOPs;

[0058] S6. Fine-tuning the model: the model is fine-tuned using the cross-modal UAV dataset two, the fine-tuned model is saved, and target detection is performed;

[0059] S7. Solidifying network parameters, saving the final target detection model.

[0060] Further, in S1, the cross-modal UAV dataset one is an AntiUAV dataset, and the cross-modal UAV dataset two is a self-made multi-modal UAV dataset.

[0061] Preprocessing includes registering unaligned images and expanding the dataset through inversion, translation, and symmetry operations.

[0062] Further, in S2, the MFFB module is divided into an infrared submodule and a visible light submodule; the infrared feature map is sequentially subjected to a large kernel sparsification operation and an SS2D operator to extract features, then the features are initially fused with the features transmitted by the visible light submodule, and after linear projection, the features are secondarily fused with the projected features of the visible light submodule to output the final multi-modal features; the visible light submodule performs a symmetry operation.

[0063] Further, the MFFB module adopts a dynamic gating mechanism to realize an environment-driven strategy through the following steps:

[0064] (a) Projecting the infrared feature map and the visible light feature map into a high-dimensional state space, respectively:

[0065]

[0066] wherein f(.) represents projecting the features into a complex state space; and represent the complex state features;

[0067] (b) Projecting and to obtain gating parameters and

[0068]

[0069] wherein, a, b represent the image adaptive adjustment operation for feature in the multi-modal task respectively;

[0070] (c) Modulating feature interaction:

[0071]

[0072] wherein, and represent the state features of visible light and infrared images after the initial feature interaction respectively; and is the element-wise product.

[0073] (d) Cross-modal deep fusion: The dual-modal features are pulled into a high-dimensional complex state space for deep fusion, and adaptive parameters are used in the cross-branch information interaction process to enable sufficient information complementarity between visible light and infrared images, obtaining complementary features and

[0074]

[0075] wherein, f out (.) represents a simple linear projection, μ1=0.64, μ2=0.49.

[0076] Further, the workflow of the dual-branch backbone network in S3 includes:

[0077] The visible light branch and the infrared branch respectively extract basic features through three-level local convolution layers;

[0078] The VSS module is used to globally enhance the basic features;

[0079] The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features are generated through residual connection to generate the next level feature map;

[0080] The level feature map is processed in two ways: one way is to input the neck network after cross-branch weighted fusion, and the other way is to pass through the local feature enhancement unit to the next level.

[0081] The two-way processing is iteratively executed twice, and the deep features are output to the neck network and the cross-modal fusion module after spatial pyramid pooling.

[0082] Further, the local convolution layer uses a C3K2 module, which includes a depth separable convolution and a channel attention mechanism; the VSS module realizes multi-scale feature stabilization through parallel hollow convolution layers and global average pooling layers.

[0083] Further, the neck network uses a bidirectional feature pyramid structure, which aggregates multi-scale features through top-down and bottom-up dual fusion methods, and embeds coordinate attention mechanisms in the fusion layer.

[0084] Embodiment 2:

[0085] A cross-modal unmanned aerial vehicle detection method based on an environment-driven strategy, comprising the following steps:

[0086] S1, preparing a data set: two cross-modal unmanned aerial vehicle data sets are prepared and the data sets are preprocessed. The first cross-modal unmanned aerial vehicle data set is used for network training and model fine-tuning, and the second cross-modal unmanned aerial vehicle data set is used for model testing;

[0087] The first cross-modal unmanned aerial vehicle data set in S1 is an infrared image unmanned aerial vehicle detection and tracking data set (Anti-UAV) in the air background. The data set is synchronously collected in dual-spectrum, each pair of data contains visible light and infrared spectrum video stream, covering 6 types of unmanned aerial vehicle targets of mainstream manufacturers such as DJI and Pano. It also contains typical airspace background elements such as buildings, clouds, and vegetation. The total size is 318 high-resolution video pairs, generating 331919 annotated frames. At the same time, each target corresponds to the label position in the image, and each image sequence corresponds to a label file.

[0088] The second cross-modal unmanned aerial vehicle data set in S1 is a self-made multi-modal unmanned aerial vehicle data set in a complex scene, which includes a small domestic unmanned aerial vehicle and a medium-sized DJI unmanned aerial vehicle. The data set includes 17341 images and 17996 targets. Geometric transformation operations such as random rotation, translation, and symmetry are performed on the images, and brightness and contrast adjustments are also performed on the images to expand the data set and improve the robustness and generalization ability in complex environments.

[0089] S2, designing a feature fusion module MFFB based on a state space model: the structure of the feature fusion module MFFB is as shown in Figure 3 The symmetric visible light and the normalized layer, linear layer, activation function, deep dilated convolution, deep convolution, channel convolution, and SS2D operator are composed. After the visible light feature map and the infrared feature map undergo the normalized layer, linear layer, and activation function, they are subjected to large kernel sparsification operation, and are sent to SS2D for cross-selective scanning. The obtained high-dimensional features are subjected to primary fusion with the multi-modal low-dimensional features of the branch, and after projection by the linear layer, they are subjected to secondary superposition deep fusion with the high-dimensional features of the other modal, and are subjected to jump connection with the initial input feature map to output the final fused features.

[0090] The MFFB module in S2 utilizes a gating mechanism to convert and fuse the dual-modal features. Among them, the VSS module is first used to project different modal features to high dimensions, and the VSS global feature extraction module is as shown in Figure 4 As shown in the following:

[0091]

[0092] where f(.) denotes the projection of the feature into the complex state space. and denote the complex state feature.

[0093] The and are also projected to obtain the gating parameters and

[0094]

[0095] where a, b denote the image adaptive adjustment operation for the feature in the multi-modal task, respectively. Then the and gating outputs are used to modulate and

[0096]

[0097] where, and denote the state feature of the visible light and infrared image after the first interaction of the feature, respectively. The · is the element-wise product.

[0098] Finally, the dual-modal features are pulled up to the high-dimensional complex state space for deep fusion, and the adaptive parameters are used in the cross-branch information interaction process to make the visible light and infrared images fully complementary in information, obtaining the complementary features and

[0099]

[0100] where f out (.) denotes a simple linear projection, and according to the experiment, μ1=0.64, μ2=0.49.

[0101] S3, constructing the MDUAV-YOLO network model: the overall structure of the model is based on the YOLOv11 network framework as shown in Figure 2 . It mainly includes three stages: single-modal feature extraction, dual-modal feature fusion, and neck and head detection. Among them, the backbone network includes a dual-branch feature extraction backbone network and three state space feature fusion blocks (MFFB), the neck network and the detection head are used for cross-modal target detection, and the neck network is as shown in Figure 5 . The feature extraction network first extracts local features from visible light and infrared images using a convolution feature extraction module (C3K2), and the C3K2 local feature extraction module is as shown in Figure 6 , denoted as and

[0102]

[0103] wherein denote the i-th layer of feature maps of visible and infrared images respectively, H, W and C denote the height, width and channel number of the feature maps respectively. RGB , I IR denote the input visible and infrared images respectively. backbone denote the feature extraction function, ψ backbone , ψ backbone denote the parameters in the feature extraction process.

[0104] By inputting the two features into the MFFB, the multi-modal features are fused in a complex state space, which reduces the difference between the cross-modal features and enhances the representation consistency of the fused features. Specifically, the two local features are first fused by the initial module of the state space fusion MFFB model. Then the interactive features are fed into the subsequent modules for deep feature fusion in a complex state space, which produces corresponding complementary features and By adding the original features and to the complementary features respectively, the local features are enhanced to form and Subsequently, the enhanced features and are directly added to generate the fused features, and the present application only adds MFFB in the last three stages to generate fused features P3, P4 and P5, which are used as the input of the neck and head of YOLOv11 to generate the final detection results.

[0105] S5, designing a loss function and determining an evaluation index: improving the composite loss function, optimizing the model multiple times so that the values of the class classification loss and the bounding box regression loss reach the expected threshold, or the iteration number reaches the preset threshold, the model ends training, and the model parameters are saved; input the test images in the data set into the network for testing, select the optimal evaluation index to measure the test results, and evaluate the precision and performance of the model.

[0106] The loss function in S5 uses a composite loss function, including a class classification loss and a bounding box regression loss;

[0107] A binary cross-entropy loss (BCE) is used to construct a multi-class discrimination mechanism, and a grid region division is performed on the input feature map to generate an N*M dimensional space anchor box. Multi-channel one-hot encoding (the encoding dimension K corresponds to the total number of target categories) is performed on each anchor box, a confidence probability distribution matrix of each channel is output through a full connection layer, and a maximum value screening strategy is implemented. BCELoss can be expressed by the following formula:

[0108]

[0109] Where N is the total number of samples, L i is the loss of the i-th sample, y i is the true label of the i-th sample, p i is the probability that the model predicts that sample i belongs to the positive class (y i =1).

[0110] The application adopts a DFL dynamic focusing mechanism (Distribution Focal Loss) and a SIoU multi-dimensional space constraint (Spatial Intersection over Union) collaborative optimization strategy. The DFL loss function establishes a discrete coordinate probability distribution model, and the target position neighborhood probability is optimized through cross-entropy loss. DFLoss can be expressed by the following formula:

[0111] L DFL (S i ,S i+1 )=-[(yi+1-y)log(S i )+(y-yi)log(S i +1)];

[0112] Where Si represents the prediction probability of the model for the adjacent discrete interval, y represents the true value of the bounding box in target detection, and y i is the adjacent reference point of the discrete interval.

[0113] Where the spatial intersection over union loss (SIoU) is coupled with multi-dimensional space features by introducing direction perception constraints, which significantly optimizes the bounding box regression accuracy. An angle cost loss function based on vector angle is established, the direction projection of the center line of the predicted box and the real box is calculated, and the problem that the traditional method fails to detect the 45° direction is solved. By introducing a diagonal standard deviation weighting factor and a nonlinear adjustment parameter, the detection accuracy after occlusion is significantly improved, especially for small target detection scenarios such as unmanned aerial vehicles. Specifically, it is composed of angle loss, distance loss, shape loss and IoU loss. SIoULoss can be expressed by the following formula:

[0114]

[0115] wherein, Delta is distance loss, Lambda is angle loss, Omega is shape loss, ch is the vertical distance between the center points of the predicted box and the real box, sigma is the standard deviation of the diagonal length of the predicted box and the real box, nu is a hyperparameter that adjusts the decay rate, and theta is a hyperparameter that adjusts the degree of nonlinearity.

[0116] The total loss in the present application can be expressed as:

[0117] L = lambda1L BCE + lambda2L DFL + lambda3L SIoU ;

[0118] wherein, in the experiment, lambda1, lambda2 and lambda3 are respectively 0.5, 1 and 5.8, and the detection effect is best.

[0119] In the evaluation index in S5, P (precision), R (recall), mAP50 (average precision of IoU threshold 0.5), mAP, FPS (frame rate), parameter quantity (Params) and GFLOPs (model calculation complexity) are used as core indexes.

[0120]

[0121]

[0122] wherein, TP represents true positive, and is the intersection over union of the predicted box and the real box of the detector greater than the IoU threshold; otherwise, it will be considered as false positive (FP). False negative (FN) means that there is a real target but it is not detected. Equation (18) shows that AP is the integral of the precision-recall curve for each class. mAP50 is the average of the AP values of all classes when IoU=0.50 is calculated in equation (17). Similarly, mAP95 calculates the average of the AP values of all classes when IoU=0.95. mAP is the main challenge index, which can be expressed as the average value when IoU=0.50:0.05:0.95.

[0123] S6, fine-tuning model: retraining and hyperparameter fine-tuning using the second infrared data set to further improve the detection network performance.

[0124] S7, save the model: after all the training is completed, fix all the parameters, and determine the multi-modal target detection model.

[0125] The present application proposes a new cross-modal target detection network architecture MDUAV-YOLO, the backbone network is designed as a dual-branch structure of visible light and infrared, combines the local detail information extraction ability of convolution and the global information extraction ability of state space model, and improves the unmanned aerial vehicle detection efficiency.

[0126] The application effectively alleviates the spatial mismatch problem of cross-modal features through the high-dimensional state evolution mechanism of the state space model, and significantly improves the deep fusion capability of multi-modal features.

[0127] The application realizes adaptive adjustment of the modal contribution degree by using a dynamic gating mechanism through the design of an environment-driven strategy, and improves the adaptability to different complex environments.

[0128] The application uses a more applicable classification loss and a bounding box regression loss to solve the problems of small UAV targets, unclear boundaries and easy confusion with the background in complex environments by designing a new composite loss function.

[0129] To verify the superiority of the method of the application, the method proposed in the application and the existing advanced method are compared in UAV detection on the data set self-made in the application, and the detection results of two scenes are as shown in Figure 7 The existing method 1 is a CFT (Cross Modality Fusion Transformer) network which captures long-range dependencies and integrates global context information by using a Transformer architecture. Its core innovation is that it can simultaneously and naturally fuse intra-modal and inter-modal features through a self-attention mechanism. This method can effectively extract the interaction information between visible light and infrared modalities, and improve the performance of multispectral target detection; the existing method 2 is an ICAFusion network which proposes a feature fusion framework with a double cross-attention mechanism. This framework uses a query-guided cross-attention mechanism to enhance the discriminability of target features. An iterative interaction mechanism is introduced to overcome the high parameter and calculation cost caused by the stacked Transformer block, and to realize parameter sharing between multi-modal Transformers at the block level. This method is universal, effective, fast in reasoning, and suitable for various practical scenarios. The cross-modal UAV detection method based on the environment-driven strategy proposed in the application can enhance the feature extraction capability by combining local detailed information and global information, and can enhance the feature fusion capability in high-dimensional space under different modalities, Figure 7 The comparison of the detection results in the

[0130] The comparative experiments performed on the same data set show that, compared with the two existing methods and the single-mode detection method, the method proposed in the application shows improvement in the cross-modal unmanned aerial vehicle target detection task in terms of relevant evaluation indexes (such as mAP, mAP50 and model calculation complexity), and specific comparison data are shown in Table 1. In the test phase, the detection method of the application has an accuracy improvement of 5.6% compared with the suboptimal method, and the calculation amount is greatly reduced. These results verify the feasibility of the method of the application and the performance advantage of the method of the application compared with the prior art.

[0131] Table 1: Comparison table of data of the application and prior art

[0132]

[0133] The above describes only the preferred specific embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A cross-modal drone detection method based on an environment-driven strategy, characterized in that: The following steps are involved: S1. Prepare datasets: Obtain and preprocess two cross-modal drone datasets. Cross-modal drone dataset 1 is used for network training, and cross-modal drone dataset 2 is used for model fine-tuning and testing. S2. Design a feature fusion module MFFB based on a state-space model: the module includes a state-space operator SS2D, a depthwise convolution, a normalization layer, a linear layer, and an activation function; S3. Build the MDUAV-YOLO network model: Build a multimodal detection framework that includes a dual-branch backbone network, a cross-modal feature fusion network, a neck network, and a head network. S4. Training the network model: training the network model using a cross-modal drone dataset; S5. Design a composite loss function and evaluation metrics: The composite loss function includes a category classification loss using the BCE loss and a bounding box regression loss using the DFL loss plus the SIoU loss. The evaluation metrics include precision (P), recall (R), mAP50, mAP, frame rate (FPS), parameter count (Params), and computational complexity (GFLOPs). S6. Fine-tune the model: Use the cross-modal drone dataset to fine-tune the model, save the fine-tuned model and perform object detection.

2. The cross-modal drone detection method based on an environment-driven strategy according to claim 1 is characterized in that: In S1, the first cross-modal drone dataset is the AntiUAV dataset, and the second cross-modal drone dataset is a self-made multi-modal drone dataset. Preprocessing includes registering misaligned images and augmenting the dataset through inversion, translation, and symmetry operations.

3. The cross-modal drone detection method based on an environment-driven strategy according to claim 1 is characterized in that: The MFFB module in S2 is divided into an infrared sub-module and a visible light sub-module; the infrared feature map is sequentially subjected to a large kernel sparsification operation and an SS2D operator to extract features, and then initially fused with the features transmitted by the visible light sub-module, and then linearly projected and then fused with the projected features of the visible light sub-module for a second time to output the final multimodal features; The visible light submodule performs symmetric operations.

4. The cross-modal UAV detection method based on an environment-driven strategy according to claim 1 or 3, characterized in that: The MFFB module adopts a dynamic gating mechanism to implement the environment-driven strategy through the following steps: (a) Project the infrared feature map and the visible light feature map into the high-dimensional state space respectively: Among them, f(.) represents the projection of features into the complex state space; and Represents complex state characteristics; (b) Projection and Get gating parameters and Among them, α and β represent the image feature adaptive adjustment operations in multimodal tasks; (c) Modulation feature interaction: in, and Represent the state features of the visible light and infrared images after the initial interaction of features; · is the element-wise product; (d) Cross-modal deep fusion: The dual-modal features are pulled up to a high-dimensional complex state space for deep fusion, and adaptive parameters are used in the cross-branch information interaction process to make the visible light and infrared images fully complementary to obtain complementary features. and Among them, f out (.) represents a simple linear projection, μ1 = 0.64, μ2 = 0.

49.

5. The cross-modal drone detection method based on an environment-driven strategy according to claim 1 is characterized in that: The workflow of the dual-branch backbone network in S3 includes: The visible light branch and infrared branch respectively extract basic features through three-level local convolutional layers; Use VSS module to globally enhance basic features; The enhanced features are input into the MFFB module for cross-modal fusion, and the fused features are connected through residual connections to generate the next level feature map; The hierarchical feature map is processed in two ways: one is weighted fusion across branches and then input into the neck network, and the other is passed to the next level through the local feature enhancement unit; The two-way processing is performed iteratively twice, and the deep features are output to the neck network and cross-modal fusion module after spatial pyramid pooling.

6. The cross-modal drone detection method based on an environment-driven strategy according to claim 5 is characterized in that: The local convolution layer adopts the C3K2 module, which includes depth-separable convolution and channel attention mechanism; the VSS module achieves multi-scale feature stabilization through parallel hollow convolution layers and global average pooling layers.

7. The cross-modal drone detection method based on an environment-driven strategy according to claim 1, characterized in that: The neck network adopts a bidirectional feature pyramid structure, aggregates multi-scale features through top-down and bottom-up dual-path fusion, and embeds a coordinate attention mechanism in the fusion layer.

Images