An event camera based end-to-end real-time small target drone detection method

Abstract

The application discloses an end-to-end real-time small target UAV detection method based on an event camera, and belongs to the technical field of UAV detection. First, an RGB image and an event stream are synchronously collected, and the event stream is processed into a spatiotemporal voxel grid. Then, the dynamic features of the event and the visual features of the RGB are respectively extracted through an SNN and a CNN double-flow network. The two kinds of multi-scale features are deeply fused through a cross-attention mechanism encoder to generate complementary enhanced features. Finally, the fused features are decoded based on a learnable object query detection head to output the accurate position and category of the UAV target in an end-to-end manner. The method can overcome the problem of low anti-UAV detection rate in various complex environments and light conditions, more effectively combine the advantages of the RGB camera and the event camera, and improve the detection performance, robustness and environmental adaptability of the UAV target.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone detection, and in particular relates to an end-to-end real-time small-target drone detection method based on an event camera. Background Art

[0002] In recent years, the rapid adoption and declining costs of small drone technology have driven the development of various industries while also creating new security challenges. Unauthorized drones can be used for illegal reconnaissance, smuggling, interference, and even attacks, posing an increasingly serious threat to low-altitude security in key areas and targets such as airports, critical infrastructure, major events, and no-fly zones. Therefore, the development of efficient and reliable counter-drone technology, particularly for the early detection, accurate identification, and continuous tracking of "low, slow, and small" targets, has become a pressing need to maintain low-altitude security and safeguard the public interest.

[0003] Drone target detection is the core link and primary prerequisite of anti-drone systems. However, target detection in anti-drone scenarios faces many unique challenges: target drones are usually small in size, fast in speed, highly maneuverable, and may have a certain degree of low detectability (stealth technology) design; at the same time, the detection system needs to operate in complex urban or outdoor backgrounds, overcoming the influence of adverse factors such as weather changes, day and night alternation, strong light interference, and background clutter; in addition, it is necessary to effectively distinguish between threatening drones and non-threatening targets such as ordinary model aircraft / birds.

[0004] Currently, RGB cameras are one of the commonly used sensors in anti-drone systems. They can provide visual images of the target, which helps in target confirmation and type recognition. The detection algorithm based on RGB images can play a certain role when the lighting is good, the target is close and the movement is relatively slow.

[0005] However, RGB cameras have significant limitations in anti-drone applications: limited detection range, poor environmental adaptability, motion blur, and background interference. To address these challenges, event cameras, as a new type of sensor, have shown potential for application in the anti-drone field due to their unique operating principles. However, existing technologies still have shortcomings in combining event cameras and RGB cameras to achieve efficient, reliable, and all-weather anti-drone target detection.

[0006] Therefore, the present invention aims to more effectively combine the advantages of RGB cameras and event cameras through a novel cross-modal fusion method, and significantly improve the target detection performance in anti-UAV scenarios. Summary of the Invention

[0007] To address the above shortcomings, the present invention provides an end-to-end real-time small target drone detection method based on event camera, which includes the following steps:

[0008] S1. Synchronous data acquisition and preprocessing:

[0009] Acquire synchronous data from a calibrated RGB camera and an event camera of the same scene, defining them as RGB image frame sequences and asynchronous event streams, respectively. Process the asynchronous event streams and convert them into spatiotemporal voxel grid representations.

[0010] S2, dual-stream multi-scale feature extraction:

[0011] Extract multi-scale RGB visual feature maps from synchronized data and multi-scale dynamic feature maps from spatiotemporal voxel grid representations;

[0012] S3. Deep fusion of cross-modal features based on hybrid encoders:

[0013] The extracted multi-scale RGB visual feature map and the multi-scale dynamic event feature map are subjected to multi-scale interaction and information complementation to generate enhanced multi-scale fusion features;

[0014] S4, target detection head decoding and output:

[0015] The target detection head detects UAV targets based on enhanced multi-scale fusion features and outputs the detection results.

[0016] Furthermore, the event voxel grid is used to encode polarity, temporal and spatial position information of events within a preset time window.

[0017] Furthermore, the multi-scale RGB visual features are extracted by a first backbone network of a deep convolutional neural network architecture, and the multi-scale dynamic features are extracted by a second backbone network of a pulse neural network architecture.

[0018] Furthermore, the multi-scale interaction and information complementation refer to:

[0019] Through the cross-attention mechanism, bidirectional guidance and enhancement are performed between multi-scale RGB visual features and multi-scale dynamic event features.

[0020] Furthermore, the target detection head includes a decoder and a detection head. The input of the decoder includes a set of learnable object queries and feature maps after passing through the encoder. Its structure is a decoder of a stacked Transformer architecture. The detection head is composed of a feedforward network. The target prediction obtained after the decoder interacts with the multi-scale fusion features is transmitted to the detection head, and the detection head detects and decodes the final detection result.

[0021] Furthermore, the two-way guidance and enhancement refers to:

[0022] Multi-scale dynamic event features are used to guide multi-scale RGB visual features to focus on dynamic areas in the scene, and multi-scale RGB visual features are used to provide semantic context for multi-scale dynamic event features to distinguish targets from background noise.

[0023] Furthermore, the first backbone network is a ConvNeXt network.

[0024] Furthermore, the preprocessing is preprocessing of the RGB image frame, including resizing, data enhancement and normalization.

[0025] Furthermore, the detection result includes the bounding box coordinates, confidence score and corresponding category label of the drone target.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. By fusing two modal data, this method solves the problem of motion blur when processing high-speed moving targets with a single RGB sensor, as well as information loss under extreme lighting conditions (such as overexposure and underexposure).

[0028] 2. To address the technical challenge of small drones having weak features in RGB images and being difficult to distinguish from the background (such as flying birds and distant buildings), this method exploits the sensitivity of event data to small brightness changes. The hybrid encoder in step S3 uses a cross-modal fusion mechanism to use event features to guide RGB features to focus on dynamic areas in the scene, providing a motion "indicator" for small targets. At the same time, RGB features provide static contextual information for event features, helping to distinguish target motion from background noise. This bidirectional enhancement mechanism can generate more discriminative fusion features compared to existing technologies, effectively reducing the missed detection rate and false alarm rate of small targets and improving the accuracy of their bounding box positioning.

[0029] 3. Unlike the simple concatenation or addition of features in existing techniques, the hybrid encoder in step S3 is able to interact with feature maps from the RGB pathway (ConvNeXt) and the event pathway (SNN) at multiple scales. Its internal cross-attention mechanism and intra-scale feature interaction module ensure that information from both modalities is fully aligned and complementary at different semantic levels (from bottom-level edges to high-level semantics). This ensures that the final fused features retain the rich appearance of RGB while incorporating the precise dynamics of events. The feature quality far exceeds that of simple fusion methods, thus supporting more accurate downstream object detection tasks.

[0030] 4. By effectively combining the RGB camera's powerful recognition capabilities in good lighting conditions with the event camera's advantages in high dynamic range and all-weather operation, while overcoming their shortcomings, the vision-based counter-UAV system can operate reliably in a wider range of environmental conditions (including day and night, different lighting conditions, and various weather factors). This enhances the all-weather and all-scenario applicability of UAVs and improves the overall reliability and deployment value of counter-UAV early warning detection.

[0031] In summary, this method can overcome the problem of low anti-UAV detection rate in various complex environments and lighting conditions, more effectively combine the advantages of RGB cameras and event cameras, and improve the detection performance, robustness and environmental adaptability of UAV targets. DETAILED DESCRIPTION

[0032] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0033] This embodiment provides an end-to-end real-time small target drone detection method based on an event camera, comprising the following steps:

[0034] S1. Synchronous data acquisition and preprocessing:

[0035] RGB cameras and event cameras deployed on UAV platforms or fixed monitoring points and accurately calibrated in time and space (including internal and external parameters and timestamp synchronization) are used to synchronously capture RGB image frame sequences and asynchronous event streams within the same field of view. The RGB image frames are then preprocessed (such as resizing to a preset resolution, performing data augmentation, and normalizing according to the requirements of the selected backbone network. It should be noted that the backbone network here includes the first backbone network and the second backbone network described below). The acquired asynchronous event stream data is accumulated and processed into a spatiotemporal voxel grid representation that can encode the polarity, temporal and spatial location information of the events within the time window according to a preset time window and spatial resolution;

[0036] S2, two-stream multi-scale feature extraction:

[0037] The first backbone network (deep convolutional neural network) based on pre-trained ConvNeXt is used to forward propagate the pre-processed RGB image frames, and multi-scale visual feature maps are extracted from its convolutional layers of different depths. These multi-scale visual feature maps are rich in the color, texture and static context information of the target. It should be noted that the ConvNeXt neural network consists of multiple stages or blocks. When processing the input image from front to back, the size of the feature map will gradually decrease, while the number of channels will increase. The convolutional layers of different depths here refer to the spatial resolution of different stages. The second backbone network based on the spike neural network (SNN) architecture is used to process the spatiotemporal voxel grid representation generated in step S1. Multi-scale dynamic feature maps are also extracted from different levels of the second backbone network. These multi-scale dynamic feature maps mainly capture the target's motion pattern, edge dynamics and change information under extreme lighting.

[0038] S3. Deep fusion of cross-modal features based on hybrid encoder:

[0039] Input the multi-scale feature maps (multi-scale visual feature maps and multi-scale dynamic feature maps) of corresponding scales extracted from the RGB path and event path in step S2 into the hybrid encoder module to obtain multi-scale fusion features;

[0040] Specifically, the RGB pathway refers to the entire processing flow starting from the input of the original RGB image frame, through preprocessing, and then through the first backbone network (ConvNeXt) for feature extraction, while the event pathway refers to the entire processing flow starting from the input of the original asynchronous event stream, through preprocessing (conversion into a spatiotemporal voxel grid), and then through the second backbone network (SNN) for feature extraction;

[0041] The hybrid encoder includes attention-based intra-scale feature interaction and a cross-modal fusion module. The attention-based intra-scale feature interaction encodes the deepest feature maps, while the cross-modal fusion module enables event features (i.e., multi-scale dynamic feature maps) to guide RGB features (i.e., multi-scale visual feature maps) to focus on dynamic areas in the scene through a cross-attention mechanism. RGB features can provide semantic context for event features, helping to distinguish true target motion from background noise. Convolution operations are used to connect feature maps of different scales to further integrate global contextual information and local details.

[0042] S4, target detection head decoding and output:

[0043] This is accomplished through a decoder and a detection head. The decoder uses a set of learnable object queries and interacts with the fused features (since the multi-scale features after the encoder are fused, the fused features are a feature map output by the encoder) to refine the target prediction corresponding to each query. The output of the decoder is then fed into the detection head, which consists of a feedforward network responsible for decoding each object query into the final detection result, that is, outputting the bounding box coordinates, confidence score and corresponding category label of the detected drone target.

[0044] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the present invention specification, or directly or indirectly applied to other related technical fields, such as the loading and unloading of other items, are included in the scope of protection of the present invention.

Claims

1. An end-to-end real-time small target drone detection method based on event camera, characterized by: The following steps are involved: S1. Synchronous data acquisition and preprocessing: Acquire synchronous data from a calibrated RGB camera and an event camera of the same scene, defining them as RGB image frame sequences and asynchronous event streams, respectively. Process the asynchronous event streams and convert them into spatiotemporal voxel grid representations. S2, dual-stream multi-scale feature extraction: Extract multi-scale RGB visual feature maps from synchronized data and multi-scale dynamic feature maps from spatiotemporal voxel grid representations; S3. Deep fusion of cross-modal features based on hybrid encoders: The extracted multi-scale RGB visual feature map and the multi-scale dynamic event feature map are subjected to multi-scale interaction and information complementation to generate enhanced multi-scale fusion features; The multi-scale interaction and information complementation mentioned above refers to: bidirectional guidance and enhancement between multi-scale RGB visual features and multi-scale dynamic event features through a cross-attention mechanism; The bidirectional guidance and enhancement refers to using multi-scale dynamic event features to guide multi-scale RGB visual features to focus on dynamic areas in the scene, and using multi-scale RGB visual features to provide semantic context for multi-scale dynamic event features to distinguish targets from background noise; S4, target detection head decoding and output: The target detection head detects UAV targets based on enhanced multi-scale fusion features and outputs the detection results.

2. The end-to-end real-time small target drone detection method based on event camera according to claim 1, characterized in that: The spatiotemporal voxel grid representation is used to encode the polarity, temporal and spatial location information of events within a preset time window.

3. The end-to-end real-time small target drone detection method based on event camera according to claim 1, characterized in that: The multi-scale RGB visual features are extracted by a first backbone network of a deep convolutional neural network architecture, and the multi-scale dynamic features are extracted by a second backbone network of a spiking neural network architecture.

4. The end-to-end real-time small target drone detection method based on event camera according to claim 1, characterized in that: The target detection head includes a decoder and a detection head. The decoder input includes a set of learnable object queries and feature maps after passing through the encoder. Its structure is a decoder with a stacked Transformer architecture. The detection head consists of a feedforward network. The target prediction obtained after the decoder interacts with the multi-scale fusion features is transmitted to the detection head, and the detection head detects and decodes it into the final detection result.

5. The end-to-end real-time small target drone detection method based on event camera according to claim 3, characterized in that: The first backbone network is a ConvNeXt network.

6. The end-to-end real-time small target drone detection method based on event camera according to claim 1, characterized in that: The preprocessing is to preprocess the RGB image frame, including resizing, data enhancement and normalization.

7. The end-to-end real-time small target drone detection method based on event camera according to claim 1, characterized in that: The detection results include the bounding box coordinates, confidence score and corresponding category label of the drone target.