A UAV Detection Method Based on Lightweight Cross-Modal Fusion

Abstract

This invention provides a lightweight cross-modal fusion-based UAV detection method, belonging to the fields of computer vision and artificial intelligence. The method includes: Step 1: multimodal data input and preprocessing; Step 2: decoupled parallel feature extraction; Step 3: multi-level feature fusion; Step 4: efficient feature aggregation; Step 5: target detection and output. Through innovative fusion and feature aggregation mechanisms, this invention significantly reduces model computational complexity while maintaining high detection accuracy, enabling real-time deployment on resource-constrained edge computing platforms such as UAVs, achieving all-weather, highly robust UAV detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for detecting unmanned aerial vehicles (UAVs) based on lightweight cross-modal fusion. Background Technology

[0002] Real-time and accurate detection of unmanned aerial vehicles (UAVs) in complex environments is an extremely challenging task due to their small size, high speed, and high maneuverability. Traditional RGB image-based detection methods suffer from severe motion blur in high-speed scenes and overexposure or underexposure under extreme lighting conditions such as strong light, low light, and high dynamic range, leading to information loss and a sharp decline in detection performance.

[0003] Event cameras, as a novel biomimetic vision sensor, can asynchronously and independently respond to pixel-level brightness changes, possessing advantages such as microsecond-level temporal resolution, ultra-high dynamic range, and low data redundancy, effectively overcoming the aforementioned challenges. Therefore, fusing the rich texture and color information of RGB images with the high temporal dynamic information of event cameras has become an effective technical approach for achieving all-weather, highly robust UAV detection.

[0004] Currently, mainstream RGB-event multimodal fusion detection solutions are mainly divided into two categories: 1. Simple Feature Fusion Scheme: For example, feature maps extracted from two modalities can be merged through simple concatenation or element-wise addition operations. Although this method has low computational cost, it fails to fully explore the deep complementary relationship between the features of the two modalities, resulting in limited fusion effect and insufficient robustness in complex scenarios.

[0005] 2. Complex Attention Fusion Schemes: For example, Transformer-based models utilize powerful cross-attention mechanisms to learn deep interactions between modalities. While these methods achieve high detection accuracy on some datasets, their complex model structures and massive computational and parameter requirements present a trade-off between performance and efficiency.

[0006] The drawbacks of existing technology are: Performance and efficiency are difficult to balance: existing technologies either use simple fusion at the expense of accuracy or use complex fusion that results in excessively high computational costs, which cannot meet the needs of drones for real-time detection at the edge.

[0007] The fusion mechanism is not robust enough: Traditional fusion methods usually treat the two modalities equally or use simple attention weighting. In specific scenarios such as extreme lighting or high-speed motion, when the information quality of one modality (such as RGB) deteriorates significantly, the model cannot dynamically and biasedly trust the other modality (such as events) with higher information quality, resulting in poor overall robustness.

[0008] Inefficient feature aggregation: When fusing multi-scale features, the information flow of the traditional Feature Pyramid Network (FPN) is not sufficient. While some improved versions perform better, the standard convolution computation in their original design has a large overhead, making them unsuitable for lightweight deployment. Summary of the Invention

[0009] This invention provides a drone detection method based on lightweight cross-modal fusion, aiming to solve the problem of difficulty in balancing performance and efficiency in existing drone detection models. It aims to significantly reduce the computational complexity of the model while maintaining high detection accuracy through innovative fusion and feature aggregation mechanisms, enabling it to be deployed in real time on resource-constrained edge computing platforms such as drones, and achieving all-weather, highly robust drone detection.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A drone detection method based on lightweight cross-modal fusion includes: Step 1: Multimodal data input and preprocessing; Receive synchronized RGB image and event data stream; preprocess the event data stream to obtain preprocessed event data; Step 2: Decoupled parallel feature extraction; The preprocessed event data and RGB images are fed into two independent and parallel backbone networks, and multi-scale feature maps are extracted through the two backbone networks respectively. Step 3: Multi-level feature fusion; Different fusion strategies are used for multi-scale feature maps: Deep semantic fusion: For the deepest feature map in the multi-scale feature map, the cross-attention mechanism is used to calibrate the features corresponding to the RGB image with the features corresponding to the preprocessed event data. Then, the contribution of the two features is balanced by the asymmetric gating mechanism, and the contribution of the features corresponding to the preprocessed event data is not lower than the preset threshold, so as to obtain a cross-modal fusion feature map. Shallow detail fusion: For feature maps of other layers in the multi-scale feature map, a stitching operation is performed to fuse them to obtain a stitched and fused feature map of different layers; Step 4: Efficient feature aggregation; By employing depthwise separable convolution, cross-scale information flow and aggregation are performed on cross-modal fusion feature maps and spliced ​​fusion feature maps from different layers through iterative top-down and bottom-up paths, resulting in enhanced feature maps. Step 5: Target detection and output; The enhanced feature map is fed into the detection head, which performs drone target location regression and classification, and outputs the detected drone bounding box and category information.

[0011] In this specification, in step 1, the event data stream is preprocessed using an efficient event filter module. The efficient event filter module learns a spatial attention map and a learnable global scalar to adaptively interpolate the input event frames to enhance motion saliency regions and suppress noise and static background, thereby obtaining preprocessed event data. The efficient event filter module learns the spatial attention map through depthwise separable convolution.

[0012] In this specification, in step 2, the backbone network is a lightweight convolutional neural network based on the YOLOv8-n model.

[0013] In this specification, in step 3, the preset threshold in the asymmetric gating mechanism is 50%.

[0014] In this specification, in step 4, a lightweight bidirectional feature pyramid network is used to perform cross-scale information flow and aggregation on cross-modal fusion feature maps and spliced ​​fusion feature maps of different layers. The lightweight bidirectional feature pyramid network contains multiple repeatedly stacked bidirectional fusion layers. The fusion node of each bidirectional fusion layer adopts a dynamic weighting strategy based on Softmax to perform weighted fusion of the input features.

[0015] In this specification, in step 5, the detection head is a YOLO detection head.

[0016] In this specification, in step 1, the RGB image and event data stream are time-synchronized RGB images and event frames.

[0017] In this specification, in step 1, the high-efficiency event filter module enhances the motion saliency region and suppresses noise and static background through adaptive interpolation to improve the signal-to-noise ratio of the event data stream.

[0018] In this specification, step 3, the mutual calibration implemented by the cross-attention mechanism specifically involves: using the dynamic information in the features corresponding to the preprocessed event data to compensate for the motion blur in the features corresponding to the RGB image, and using the texture information in the features corresponding to the RGB image to enrich the sparse representation in the features corresponding to the preprocessed event data.

[0019] In this specification, in step 4, the lightweight bidirectional feature pyramid network uses depthwise separable convolution, which significantly reduces the amount of computation and parameters while fusing high-level semantic information and low-level spatial details in the enhanced feature map to cope with the scale changes of the UAV in three-dimensional space.

[0020] In summary, the present invention has at least the following beneficial effects: This method achieves an excellent balance between performance and efficiency: by employing a lightweight backbone network, depthwise separable convolutions, and an efficient fusion module, it maintains high detection accuracy (0.932 mAP50 and 0.573 mAP50:95 on the NeRDD dataset) while having only 6.9M model parameters. This resolves the contradiction in existing technologies where high-precision models are too large and lightweight models lack sufficient accuracy.

[0021] Significantly improved model robustness: The asymmetric gating mechanism in the core CGF module ensures that even in extreme scenarios where RGB image quality is severely degraded (such as strong light overexposure and high-speed motion blur), the model can still make reliable judgments based on high-quality event data, greatly enhancing the robustness of the detection system in all weather and complex environments.

[0022] This method meets the requirements for real-time deployment at the edge: due to its lightweight overall architecture, it has low computational complexity and fast inference speed. It can meet the application needs of real-time detection on UAVs and has high practical value. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the UAV detection method based on lightweight cross-modal fusion involved in this invention.

[0025] Figure 2 This is a schematic diagram of the overall network structure involved in this invention.

[0026] Figure 3 This is a schematic diagram of the EEF module involved in this invention.

[0027] Figure 4 This is a schematic diagram of the CGF module involved in this invention.

[0028] Figure 5 This is a schematic diagram of the Lit-BiFPN module involved in this invention. Detailed Implementation

[0029] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0030] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0031] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] like Figure 1 As shown, this embodiment provides a UAV detection method based on lightweight cross-modal fusion, including: Step 1: Multimodal data input and preprocessing; Receive synchronized RGB image and event data stream; preprocess the event data stream to obtain preprocessed event data; Step 2: Decoupled parallel feature extraction; The preprocessed event data and RGB images are fed into two independent and parallel backbone networks, and multi-scale feature maps are extracted through the two backbone networks respectively. Step 3: Multi-level feature fusion; Different fusion strategies are used for multi-scale feature maps: Deep semantic fusion: For the deepest feature map in the multi-scale feature map, the cross-attention mechanism is used to calibrate the features corresponding to the RGB image with the features corresponding to the preprocessed event data. Then, the contribution of the two features is balanced by the asymmetric gating mechanism, and the contribution of the features corresponding to the preprocessed event data is not lower than the preset threshold, so as to obtain a cross-modal fusion feature map. Shallow detail fusion: For feature maps of other layers in the multi-scale feature map, a stitching operation is performed to fuse them to obtain a stitched and fused feature map of different layers; Step 4: Efficient feature aggregation; By employing depthwise separable convolution, cross-scale information flow and aggregation are performed on cross-modal fusion feature maps and spliced ​​fusion feature maps from different layers through iterative top-down and bottom-up paths, resulting in enhanced feature maps. Step 5: Target detection and output; The enhanced feature map is fed into the detection head, which performs drone target location regression and classification, and outputs the detected drone bounding box and category information.

[0033] In some embodiments, in step 1, the event data stream is preprocessed using an efficient event filter module. The efficient event filter module performs adaptive interpolation on the input event frames by learning a spatial attention map and a learnable global scalar to enhance motion saliency regions and suppress noise and static background, thereby obtaining preprocessed event data. The efficient event filter module learns the spatial attention map through depthwise separable convolution.

[0034] In some embodiments, in step 2, the backbone network is a lightweight convolutional neural network based on the YOLOv8-n model.

[0035] In some embodiments, in step 3, the preset threshold in the asymmetric gating mechanism is 50%.

[0036] In some embodiments, in step 4, a lightweight bidirectional feature pyramid network is used to perform cross-scale information flow and aggregation on cross-modal fusion feature maps and spliced ​​fusion feature maps of different layers. The lightweight bidirectional feature pyramid network contains multiple repeatedly stacked bidirectional fusion layers. The fusion node of each bidirectional fusion layer uses a dynamic weighting strategy based on Softmax to perform weighted fusion of the input features.

[0037] In some embodiments, in step 5, the detection head is a YOLO detection head.

[0038] In some embodiments, in step 1, the RGB image and event data stream is a time-synchronized RGB image and event frame.

[0039] In some embodiments, in step 1, the efficient event filter module enhances the motion saliency region and suppresses noise and static background through adaptive interpolation to improve the signal-to-noise ratio of the event data stream.

[0040] In some embodiments, in step 3, the mutual calibration implemented by the cross-attention mechanism specifically involves: using the dynamic information in the features corresponding to the preprocessed event data to compensate for the motion blur in the features corresponding to the RGB image, and using the texture information in the features corresponding to the RGB image to enrich the sparse representation in the features corresponding to the preprocessed event data.

[0041] In some embodiments, in step 4, the lightweight bidirectional feature pyramid network employs depthwise separable convolution, which significantly reduces the amount of computation and parameters while fusing high-level semantic information and low-level spatial details in the enhanced feature map to cope with the scale changes of the UAV in three-dimensional space.

[0042] The technical concept of this invention is as follows: like Figure 2 As shown, the core of this invention is a lightweight fusion architecture based on a two-stream neural network. The method includes the following steps: Step S1: Multimodal data input and preprocessing It receives synchronized RGB image and event data streams. The event data stream first passes through an Efficient Event Filter (EEF) module (such as...). Figure 3 The module performs preprocessing on the input event frames (as shown). It uses depthwise separable convolution to learn a spatial attention map and combines it with a learnable global scalar to adaptively interpolate the input event frames. The goal is to enhance motion-salient regions while suppressing noise and static background, thereby improving the signal-to-noise ratio of event data with extremely low computational cost.

[0043] Step S2: Decoupled Parallel Feature Extraction The preprocessed event data and the original RGB image are fed into two independent, parallel backbone networks. This decoupled design allows the networks to extract multi-scale feature maps, such as P2, P3, P4, and P5 levels, for the different characteristics of the two modalities (the texture of RGB and the dynamics of the event).

[0044] Step S3: Multi-level feature fusion Different fusion strategies are used for feature maps at different levels: Deep semantic fusion: On the highest-level feature map (P5) with the richest semantic information, an innovative cross-modal gated fusion (CGF) module (such as...) is used. Figure 4(As shown). This module first uses a cross-attention mechanism to calibrate RGB features and event features, that is, using the dynamic information of events to compensate for the motion blur of RGB, and using the texture information of RGB to enrich the sparse representation of events. Then, a final fusion is performed through an asymmetric gating mechanism. This mechanism learns a dynamic weight gate to balance the contributions of the two modalities while introducing a bias on the event flow. Specifically, even if the gating weights are completely biased towards RGB features, the contribution of event features is still guaranteed to be no less than a preset threshold (e.g., 50%), as a kind of "fault protection" mechanism to ensure that when RGB information fails, the key motion information is never completely suppressed, thereby improving the overall robustness of the model.

[0045] Shallow detail fusion: A simple concatenation operation is used to fuse shallower feature maps (P2, P3, P4). This design aims to preserve high-frequency spatial details such as the drone's outline and rotors, while avoiding unnecessary computational overhead at lower feature abstraction levels.

[0046] Step S4: Efficient Feature Aggregation All the multi-scale feature maps (P2, P3, P4, P5) fused in step S3 are fed into a lightweight bidirectional feature pyramid network (Lit-BiFPN, e.g.) Figure 5 (As shown). This network achieves efficient information flow across scales through iterative top-down and bottom-up paths. To achieve lightweight design, all standard convolutions in the network are replaced with depthwise separable convolutions, which significantly reduces computational cost and parameter count while effectively fusing high-level semantic information with low-level spatial details to cope with the huge scale changes of UAVs in three-dimensional space.

[0047] Step S5: Target Detection and Output The enhanced feature map output by Lit-BiFPN is fed into the detection head for target location regression and classification, and finally the detected drone bounding box and category information are output.

[0048] Figure 2In this architecture, Conv, C2f, SPPF, C2PSA, and Detect are modules included in the standard YOLO structure. Conv is a standard convolutional module used for feature extraction and channel adjustment. C2f is a lightweight feature fusion module that achieves efficient fusion of shallow and deep features through cross-stage partial connections, improving the network's gradient flow and feature reuse capabilities. SPPF (Spatial Pyramid Pooling-Fast) is a spatial pyramid pooling acceleration module that extracts contextual information from different receptive fields through multi-scale pooling operations to enhance the multi-scale expressiveness of features. C2PSA is an improved attention fusion module that combines the C2f structure with the PSA (Parallel Spatial Attention) mechanism, achieving adaptive feature weighting in spatial dimensions while maintaining computational efficiency, thereby enhancing the feature response of salient regions. Detect is the detection head module responsible for mapping the fused multi-scale features to specific prediction tasks, including target location (bounding box), class probability, and confidence, to achieve the final target detection output. Efficient Event Filter, CGF, and Lit-BiFPN are innovative concepts proposed in this invention.

[0049] Figure 3 In this module, Depthwise Conv (5×5) is a channel-wise convolution module used to extract local spatial features while maintaining computational efficiency and reducing the number of parameters. BatchNorm is a batch normalization module used to accelerate network convergence and improve training stability. ReLU is a non-linear activation function module used to enhance the feature representation capability of the network. 1×1 Conv is a channel transformation module used for feature compression and channel fusion to reduce computational overhead. Sigmoid is an activation function module used to map features to the [0,1] interval to generate an attention weight map. Attention Map is an attention map generation module that highlights key regions through feature weighting to improve the discriminative ability of features. Residual Fusion is a residual fusion module that fuses input features with attention-modulated features to achieve information preservation and enhancement. Output is the output module that outputs the attention-enhanced feature map.

[0050] Figure 4In this module, the 1×1 Conv module performs linear mapping and channel alignment of RGB and DVS features, reducing the computational complexity of subsequent fusion. The CrossAttention module establishes dependencies between RGB and DVS features, enabling selective interaction and enhancement of complementary information. The SpatialAttention module highlights salient regions and suppresses redundant background features in the spatial dimension. The ChannelAttention module strengthens key feature channels through channel weight learning, improving feature representation. The Concat module fuses the attention-enhanced RGB and DVS features along the channel dimension, providing a joint representation for subsequent convolutions. The Fuse module, composed of 1×1 and 3×3 convolutions, fuses cross-modal features and integrates local and global contextual information to generate an initial fused feature map. The Gate module, consisting of 1×1 convolutions, SiLU activation, 1×1 convolutions, and a Sigmoid function, adaptively adjusts the weights of fused features and event features to achieve dynamic feature balance. The Output module performs weighted combination based on the gating weights and outputs the final fused feature map, achieving effective complementarity between RGB and event modality.

[0051] Figure 5 In the diagram, P2, P3, P4, and P5 Inputs represent multi-scale feature inputs, signifying feature maps from different layers of the backbone network. P5 contains the strongest semantic information but has the lowest resolution, while P2 contains the richest spatial detail and has the highest resolution. Weighted Fusion is a weighted fusion module used to merge features from different paths (original input, top-down, and bottom-up). It adaptively learns the importance of each input feature and performs weighted summation, thus efficiently integrating cross-scale information. DWSconv is a Depthwise Separable Convolution module, serving as an efficient feature processing unit. Modules with the -td suffix (e.g., P2-td) are located in the top-down path and are used to process and pass high-level semantic information. Modules with the -out suffix (e.g., P2-out) are used for the final processing of the bidirectionally fused features, generating the outputs for each layer. P2-out, P3-out, P4-out, etc., represent the final multi-scale feature maps of the output. These feature maps have undergone bidirectional information flow and fusion along both top-down and bottom-up paths, thus containing both rich semantic information and precise localization details, making them suitable for subsequent tasks.

[0052] In one specific embodiment, this invention proposes a two-stream neural network model, named MultiEdgeNet, specifically designed for real-time UAV target detection at the edge. This model is based on the Ultralytics YOLO framework and introduces core innovations: An Efficient Event Filter (EEF) module designed for event data improves the signal-to-noise ratio of event data with extremely low computational cost; A cross-modal gated fusion module (CGF) enables deep semantic collaboration between RGB and event features through a novel asymmetric gating mechanism. A lightweight bidirectional feature pyramid network (Lit-BiFPN) efficiently aggregates multi-scale features to robustly detect drones with extreme scale variations.

[0053] MultiEdgeNet aims to efficiently integrate texture information from RGB cameras with high dynamic range information from event cameras, especially in challenging scenarios such as high-speed motion and complex lighting, achieving a balance between high precision and high efficiency. The specific process is as follows: Step S1: Data Input and Efficient Preprocessing The input data consists of time-synchronized RGB images and event frames. The event stream is first processed by the EEF module. The specific implementation of this module is as follows: This module first efficiently learns and generates a single-channel spatial attention map A from the input event frame E using a network composed of depthwise separable convolutions. Subsequently, the filtered event frame E′ is generated through a novel adaptive interpolation mechanism, the process of which is defined as follows: ; in It represents the element-wise product (Hadamard Product). It is a learnable scalar parameter. It is a spatial attention map. The brilliance of this formula lies in its dual adaptability: in the spatial dimension, the attention map A makes the region of high motion saliency ( The original signal is preserved, while low-significance regions such as noise are preserved. Signals from the data are effectively suppressed; at the global level, the learnable parameter α allows the network to dynamically learn the optimal filtering strength based on the overall statistical characteristics of the data. Therefore, EEF provides a high-quality event stream with dynamic noise reduction and signal enhancement for the subsequent backbone network with almost no increase in computational overhead, laying the foundation for the final high-performance detection.

[0054] Step S2: Parallel multi-scale feature extraction The RGB image and the event frame E′ processed by the EEF module are fed into two parallel, lightweight CNN backbone networks based on the YOLOv8-n model. The two networks independently extract four feature maps at different scales {P2, P3, P4, P5}. This decoupled design ensures that the unique characteristics of the two modalities are fully represented and learned before fusion.

[0055] Step S3: Cross-modal feature fusion To fully leverage the complementary advantages of RGB and event data, this invention introduces a CGF module at the deepest feature scale (P5) where semantic information is richest. This module aims to address a fundamental challenge: how to dynamically calibrate and co-enhance the rich but motion-blurred texture information in RGB images with the sparse but motion-capturing cues in event streams. This process consists of three stages: (1) Cross-Modal Recalibration: First, the RGB features are... and event characteristics and We obtain by projecting independent 1x1 convolutions into a shared embedding space. and Subsequently, the multi-head cross-attention mechanism allows the two modalities to "query" each other to inject contextual information. For example, event-enhanced RGB features... It is by... As a query (Query, Q), It is computed as a key (Key, K) and a value (Value, V), supplemented by a residual join: ; This step enables the dynamic calibration and compensation of motion blur in RGB features at corresponding spatial locations for high temporal resolution motion cues in the event stream, which is particularly crucial for detecting fast-moving drones. Symmetrically, the rich texture information of RGB is also used to enhance the contextual representation of event features, resulting in... .

[0056] (2) Intra-Modal Refinement: After cross-modal calibration, the two enhanced feature maps are obtained. and The signal is sequentially refined through spatial attention and channel attention modules. This step aims to suppress irrelevant noise within each modality, purify the signal, and amplify the most informative feature responses, ultimately yielding the refined features. and .

[0057] (3) Asymmetric Gated Fusion: The final fusion is dominated by a dynamic, data-driven gating mechanism that introduces a novel asymmetric bias to the event stream. First, a spatially variable gating graph... From the refined features after splicing The learned G, as an adaptive "confidence map," dynamically modulates the fused features. (Depend on and (Merged from fusion). Unlike traditional gated fusion, the fusion output O of this invention is calculated as follows: ; This process introduces an asymmetric residual bias oriented towards event features. The weights of the event features are... Its value range is [0.5, 1). This means that even in the fully trusted fusion feature of gated G, The region (G→1), the original event characteristics It still contributes at least 50% weight to the final output. This design is based on a key prior: event data is highly robust in challenging scenarios such as high-speed maneuvering or extreme lighting. The asymmetric bias of this invention ensures that this key information is never completely covered during the fusion process, thus providing the model with a "fail-safe" mechanism to enhance robustness.

[0058] Unlike complex fusion strategies for deep semantics, this invention employs an efficient direct concatenation operation to aggregate RGB and event features at shallower feature scales {P2, P3, P4}. This design aims to preserve high-frequency spatial details such as drone outlines and rotors without loss, avoiding unnecessary computational overhead at stages where feature abstraction is low.

[0059] Step S4: Lightweight Feature Aggregation The series of multi-scale feature maps obtained by fusing in step S3 are fed into the Lit-BiFPN module. This module consists of multiple repeatedly stacked bidirectional fusion layers.

[0060] Efficient bidirectional information flow: Lit-BiFPN establishes a high-bandwidth information flow between feature maps of different scales through iterative top-down and bottom-up paths, effectively fusing high-level semantics with low-level spatial details.

[0061] Adaptive Weighted Fusion: At each fusion node of BiFPN, this invention employs a dynamic weighting strategy based on Softmax to fuse features from different paths. For a node receiving N input features... The nodes, whose fusion process is governed by a set of learnable weights. Control, specifically calculated as follows: ; This adaptive weighting mechanism allows the network to dynamically identify and enhance the most informative feature scales, which is particularly crucial for detecting drone targets with significant scale variations.

[0062] Computational efficiency: To ensure the network can run in real-time on resource-constrained edge platforms, all convolutional operations within Lit-BiFPN are implemented using lightweight depthwise separable convolution (DWSConv), the structure of which can be represented as: ; in, Represents depthwise convolution. This represents pointwise convolution, which has a computational cost far lower than standard convolution.

[0063] In summary, Lit-BiFPN, as the key hub in the architecture of this invention, solves the inherent problems of feature heterogeneity and target scale variability after multimodal feature fusion with extremely high computational efficiency.

[0064] Step S5: Detection Head and Output Finally, the feature map aggregated by Lit-BiFPN is fed into a standard YOLO detection head, which is responsible for predicting the bounding box location, confidence level, and category of the UAV.

[0065] Experimental verification and effect analysis To verify the feasibility and beneficial effects of the present invention, a series of experiments were conducted.

[0066] Training platform: NVIDIA 4090 GPU Dataset: NeRDD, a public and challenging RGB-based benchmark dataset for event drone detection.

[0067] Evaluation metrics: mAP50, mAP50:95, Parameters(M).

[0068] Experiment 1: Performance Comparison with Existing Technologies This invention compares the method of this invention (labeled Ours) with several mainstream unimodal and multimodal detection methods on the NeRDD dataset.

[0069] ; analyze: 1. Validity: The results show that fusing RGB and event data (YOLO-Fuse, Ours) outperforms any single-modal method (YOLOV8, RT-DETR), demonstrating the necessity of multimodal fusion.

[0070] 2. Superiority: The method of this invention outperforms all other methods, including the existing fusion method YOLO-Fuse, in both the core accuracy metrics of mAP50 and mAP50:95. This demonstrates the superiority of the CGF and Lit-BiFPN modules proposed in this invention in feature fusion and aggregation.

[0071] 3. Lightweight: Despite having the highest accuracy, the number of parameters in the method of this invention (6.9M) is much smaller than that of complex models (such as 42.9M of RT-DETR) and is on the same order of magnitude as other lightweight models, which proves its efficiency.

[0072] Experiment 2: Ablation Study (Verifying the Contribution of Each Module) To verify the effectiveness of the various innovative modules (EEF, CGF, Lit-BiFPN) in this invention, ablation experiments were conducted. The baseline model was a YOLOv8 two-stream network using simple splicing and fusion.

[0073] ; Analysis: Each step of the ablation experiment resulted in performance improvement, clearly demonstrating the effectiveness of the three core modules proposed in this invention—EEF, CGF, and Lit-BiFPN—and their synergistic effect when combined.

[0074] In summary, this specific embodiment, through detailed step descriptions and ample experimental data, demonstrates the feasibility and advancement of the lightweight cross-modal fusion method proposed in this invention. It not only surpasses existing technologies in detection accuracy, but more importantly, its high computational performance enables real-time, robust UAV detection on resource-constrained UAV platforms, solving key problems existing in the background technology.

[0075] Explanation of the English abbreviations and codes that appear: UAV: Unmanned Aerial Vehicle.

[0076] RGB: Red, Green, Blue, refers to standard visible light color images.

[0077] DVS: Dynamic Vision Sensor, also known as an event camera.

[0078] EEF: Efficient Event Filter. This invention proposes a preprocessing module.

[0079] CGF: Cross-modal Gated Fusion. This is the core fusion module proposed in this invention.

[0080] BiFPN: Bidirectional Feature Pyramid Network.

[0081] Lit-BiFPN: Lightweight BiFPN, a lightweight BiFPN proposed in this invention.

[0082] DWSConv: Depthwise Separable Convolution. An efficient convolution operation.

[0083] CNN: Convolutional Neural Network.

[0084] mAP stands for Mean Average Precision. It is a core metric for measuring model accuracy in object detection tasks. mAP50 refers to the mAP with an IoU threshold of 0.5, and mAP50:95 refers to the average mAP with an IoU threshold ranging from 0.5 to 0.95, with a step size of 0.05.

[0085] FPS: Frames Per Second. A metric for measuring the inference speed of a model.

[0086] YOLO: You Only Look Once, a mainstream real-time object detection framework.

[0087] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0088] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0089] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0090] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0091] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0092] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0093] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0094] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0095] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0096] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A UAV detection method based on lightweight cross-modal fusion, characterized in that, include: Step 1: Multimodal data input and preprocessing; Receive synchronized RGB image and event data streams; Preprocessing the event data stream yields preprocessed event data; Step 2: Decoupled parallel feature extraction; The preprocessed event data and RGB images are fed into two independent and parallel backbone networks, and multi-scale feature maps are extracted through the two backbone networks respectively. Step 3: Multi-level feature fusion; Different fusion strategies are used for multi-scale feature maps: Deep semantic fusion: For the deepest feature map in the multi-scale feature map, the cross-attention mechanism is used to calibrate the features corresponding to the RGB image with the features corresponding to the preprocessed event data. Then, the contribution of the two features is balanced by the asymmetric gating mechanism, and the contribution of the features corresponding to the preprocessed event data is not lower than the preset threshold, so as to obtain a cross-modal fusion feature map. Shallow detail fusion: For feature maps of other layers in the multi-scale feature map, a stitching operation is performed to fuse them to obtain a stitched and fused feature map of different layers; Step 4: Efficient feature aggregation; By employing depthwise separable convolution, cross-scale information flow and aggregation are performed on cross-modal fusion feature maps and spliced ​​fusion feature maps from different layers through iterative top-down and bottom-up paths, resulting in enhanced feature maps. Step 5: Target detection and output; The enhanced feature map is fed into the detection head, which performs drone target location regression and classification, and outputs the detected drone bounding box and category information.

2. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 1, the event data stream is preprocessed using an efficient event filter module. The efficient event filter module learns a spatial attention map and a learnable global scalar to adaptively interpolate the input event frames, thereby enhancing motion saliency regions and suppressing noise and static background to obtain preprocessed event data. The efficient event filter module learns the spatial attention map through depthwise separable convolution.

3. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 2, the backbone network is a lightweight convolutional neural network based on the YOLOv8-n model.

4. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 3, the preset threshold in the asymmetric gating mechanism is 50%.

5. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 4, a lightweight bidirectional feature pyramid network is used to perform cross-scale information flow and aggregation on cross-modal fusion feature maps and spliced ​​fusion feature maps from different layers. The lightweight bidirectional feature pyramid network contains multiple stacked bidirectional fusion layers. The fusion nodes of each bidirectional fusion layer use a dynamic weighting strategy based on Softmax to perform weighted fusion of the input features.

6. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 5, the detection head is a YOLO detection head.

7. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 1, the RGB image and event data stream are time-synchronized RGB image and event frames.

8. The UAV detection method based on lightweight cross-modal fusion according to claim 2, characterized in that, In step 1, the high-efficiency event filter module enhances the motion saliency region and suppresses noise and static background through adaptive interpolation to improve the signal-to-noise ratio of the event data stream.

9. The UAV detection method based on lightweight cross-modal fusion according to claim 1, characterized in that, In step 3, the mutual calibration achieved by the cross-attention mechanism is specifically as follows: using the dynamic information in the features corresponding to the preprocessed event data to compensate for the motion blur in the features corresponding to the RGB image, and using the texture information in the features corresponding to the RGB image to enrich the sparse representation in the features corresponding to the preprocessed event data.

10. The UAV detection method based on lightweight cross-modal fusion according to claim 5, characterized in that, In step 4, the lightweight bidirectional feature pyramid network adopts depthwise separable convolution, which significantly reduces the amount of computation and parameters while fusing high-level semantic information and low-level spatial details in the enhanced feature map to cope with the scale changes of the UAV in three-dimensional space.

Images