An unmanned aerial vehicle target detection and tracking method based on event stream vision

Abstract

This invention proposes a UAV target detection and tracking method based on event stream vision, relating to the fields of image processing and target detection technology. By modeling the spatiotemporal sparsity of event data, it solves the problems of low detection accuracy and insufficient real-time performance of existing visual detection technologies under complex lighting conditions and fast-moving scenarios. This method achieves efficient feature representation through event stream time window construction and noise suppression mechanisms, and combines a lightweight detection network with a motion-correlation-based online tracking strategy, significantly improving the detection rate of small targets while ensuring low latency and low power consumption. This method has advantages such as strong adaptability and low resource consumption, and can be used for long-term monitoring and rapid early warning of low-altitude, slow-moving UAVs.

Description

Technical Field

[0001] This invention relates to the fields of image processing and target detection technology, and specifically to a method for UAV target detection and tracking based on event flow vision. Background Technology

[0002] With the continuous increase in the density of activities in low-altitude airspace and the growing potential threats posed by small drones in scenarios such as public safety, key area protection, and military-civilian integration, real-time detection and tracking capabilities with a wide field of view have become an important technical requirement for low-altitude monitoring systems. Stable and accurate perception of small drones under conditions of complex backgrounds, extreme lighting, and rapid maneuvering is of great significance for improving low-altitude security and control capabilities.

[0003] Existing detection methods for small drones typically rely on traditional frame-based vision, millimeter-wave radar, or infrared imaging. Frame-based vision offers advantages such as ease of deployment, low cost, and rich information content, but it is limited by a fixed frame rate, easily resulting in motion blur in high-speed moving scenes, and its image quality deteriorates significantly in strong light, backlight, and low-light environments. Deep learning-based visual detection algorithms perform well in conventional images, but their performance degrades significantly in low-altitude scenes with sparse targets, small targets, and complex backgrounds. Furthermore, traditional image sensing modes generate a large amount of redundant data, making real-time inference difficult on resource-constrained edge devices. Although some studies have attempted to improve this problem using lightweight network structures, motion compensation, or multimodal fusion, stability and real-time performance remain difficult to balance in high dynamic range, low event density, and complex interference scenarios.

[0004] In summary, existing methods struggle to achieve real-time, high-precision target detection and tracking in scenarios with complex backgrounds, extreme lighting, and varying target scales. Summary of the Invention

[0005] This invention addresses the challenge of achieving stable detection and continuous tracking of neuromorphic event data in complex backgrounds, extreme lighting conditions, and scenarios with small UAV targets. It proposes an event-stream vision-based method for UAV target detection and tracking. By constructing event frames with fixed time windows, introducing a lightweight detection network adapted to the visual characteristics of events, combining an event-driven online tracking strategy, and a detection-tracking result fusion mechanism, this invention achieves efficient representation of sparse event information and real-time, robust perception of high-speed, maneuverable small UAVs. This effectively improves target detection accuracy and tracking stability, meeting the requirements of low-power, low-latency engineering applications.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for UAV target detection and tracking based on event flow vision includes the following steps:

[0008] Step 1: Obtain the raw event stream data output by the neuromorphic sensor, and use a fixed time window method to continuously segment the events to construct event frames;

[0009] Step 2: Perform polarity filtering and spatial and temporal voxel encoding on the constructed event frames to extract key change information from the event data and form a salient feature representation of the weak UAV target;

[0010] Step 3: A lightweight detection model based on the visual characteristics of events is used to detect possible drone target regions in the event frame and obtain the position and confidence output of candidate targets.

[0011] Step 4: Employ an event-driven online target tracking model to perform real-time correlation and position updates on detected candidate targets based on the target's historical state, event temporal distribution, and motion patterns, thereby achieving stable tracking output.

[0012] Step 5: Fuse the detection and tracking results, and perform confidence correction on the results based on the event trigger intensity and target motion continuity to obtain the final UAV detection and tracking trajectory.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0014] (1) Achieved robust perception capability in high dynamic scenes: The present invention adopts an event-driven data expression method, which effectively avoids motion blur and exposure saturation problems in traditional frame images through high temporal resolution event stream input, so that the system can still maintain stable perception capability in complex environments such as strong light backlight, weak light, and strong contrast background.

[0015] (2) Significantly improves the detection performance of small UAV targets: This invention enables the model to extract more discriminative dynamic features from sparse event data through spatiotemporal voxelization coding, multi-channel event feature modeling and lightweight detection structure optimized for small targets, thereby effectively enhancing the detection accuracy of long-distance, small-scale, low-contrast UAVs.

[0016] (3) Real-time processing of integrated detection and tracking: The present invention constructs an online tracking mechanism based on event continuity, realizes rapid association and update of target status on the basis of detection output, and greatly reduces processing latency; combined with a lightweight network structure, the whole system can still run in real time on resource-constrained embedded platforms.

[0017] (4) Improved robustness and reliability of the system under complex backgrounds: This invention can effectively filter background event interference such as wind-blown leaves and cloud changes by integrating event noise suppression, background interference filtering and temporal consistency of detection and tracking results, reduce false detection and drift, and make the UAV trajectory output more stable and reliable. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of an event flow vision-based UAV target detection and tracking method according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0020] like Figure 1 As shown, the present invention provides a method for UAV target detection and tracking based on event flow vision, comprising the following steps:

[0021] Step 1: Obtain the raw event stream data output by the neuromorphic sensor, and use a fixed time window method to continuously segment the events to construct event frames.

[0022] In step 1, the events output by the neuromorphic sensor Composed of quaternions:

[0023] ;

[0024] in, Represents pixel coordinates, For the event Occurrence timestamp, This indicates the polarity of the brightness change. Due to the sparse, asynchronous, and high temporal resolution characteristics of event stream data, this invention employs a fixed time window to facilitate processing by deep learning models. The event stream data is segmented, and events within a continuous time window are aggregated into a single event frame. :

[0025] ;

[0026] at the same time, To address the issue of background noise events and to suppress window indexing, this invention further employs a joint filtering strategy combining event density threshold and timestamp consistency. Specifically, for each newly arrived event... In the Centered on, with a spatial radius of Historical events are searched within the neighborhood of each pixel, and a time surface matrix is ​​introduced to record the timestamp of the most recent event for each pixel. A time consistency function is defined:

[0027] ;

[0028] in, The minimum time difference between the current event and all neighboring events. For the event The set of neighbor events, Timestamps of previous events within the neighborhood. This is a preset time-related threshold. If the time difference between the current event and its nearest neighboring event is less than... If this is the case, the event is considered to be a correlated event caused by the target's motion, rather than random noise. Within a fixed time window... Within this scope, the distribution density of events that satisfy the time consistency check is statistically analyzed in the local space. The judgment criterion is defined as follows: if the time correlation threshold is met... Within a local neighborhood, the total number of associated events exceeds the density threshold. If the event is positive, retain it; otherwise, classify it as an isolated noise event and discard it. Time-related threshold. and density threshold The settings are primarily based on target distance and movement speed. After being filtered by the aforementioned joint strategy, the valid event stream data is remapped to event frames according to its original polarity and spatiotemporal coordinates. middle.

[0029] This strategy effectively filters out thermal noise and low-frequency background fluctuations caused by dark current in neuromorphic sensors. The processed event frames maintain the high dynamic range and high temporal accuracy of event vision, while also possessing a two-dimensional image structure, facilitating subsequent deep feature extraction by the model.

[0030] Step 2: Perform polarity filtering and space-time voxelization encoding on the constructed event frames to extract change information from the event data and form a salient feature representation of weak UAV targets.

[0031] In step 2, for each event frame First, perform channel-specific statistics based on event polarity:

[0032] ;

[0033] ;

[0034] Subsequently, the time dimension is divided into B equally spaced sub-intervals, a three-dimensional voxel grid is constructed, and the events are spatiotemporally encoded:

[0035] ;

[0036] in, This represents the voxel intensity value. For the b-th time interval, This is the indicator function. This encoding method projects sparse events onto a multi-channel pseudo-image structure, explicitly representing the temporal distribution, local activity density, and polarity changes of the events. To further enhance the local contrast of weak UAV targets, this invention introduces local density normalization and spatial neighborhood aggregation operations after voxel encoding:

[0037] ;

[0038] in, This is the output after normalization. It represents the pixel neighborhood region.

[0039] This operation can effectively suppress interference from high-contrast backgrounds such as the sky and clouds, highlighting the spatiotemporal structural features of the UAV target.

[0040] Through the above preprocessing, the event frames are transformed into multi-channel input features with rich spatiotemporal variation information, providing reliable and highly discriminative input data for subsequent lightweight detection models.

[0041] Step 3: A lightweight detection model based on the visual characteristics of events is used to quickly detect the potential UAV target regions in the feature representation obtained in Step 2, and to obtain the position and confidence output of the candidate targets.

[0042] In step 3, a lightweight detection network adapted to the visual characteristics of events is used to perform forward reasoning on the feature representation constructed in step 2, extracting region information that may contain UAV targets. The detection model consists of three components: a backbone network, a feature fusion layer, and a decoupled detection head.

[0043] First, the feature representation obtained in step 2 is used as the feature map. Input the backbone network. The backbone network employs a lightweight structure composed of a hybrid of depthwise separable convolutions and deformable convolutions, extracting semantic features at different scales layer by layer to obtain multi-level outputs:

[0044] ;

[0045] in, This represents the i-th feature extraction module based on a convolutional layer. These are semantic features at different levels. It represents the deepest features with the strongest semantics and is suitable for capturing motion structure information in complex backgrounds.

[0046] Subsequently, the feature fusion layer (Neck) performs bidirectional fusion of the multi-scale features extracted by the backbone network, both top-down and bottom-up. The top-down path upsamples and concatenates deep and shallow features, enhancing the model's ability to locate small-scale targets. The bottom-up path downsamples and backsamples the fused features, ensuring that the feature map retains high-frequency motion edge features of the event flow while possessing strong semantics. The fused multi-scale features... It can be recorded as:

[0047] ;

[0048] in, This represents the fusion operator. The final output is a fused feature map at three scales.

[0049] Finally, the detection head performs classification and regression prediction on each layer of fused features, and obtains the candidate target location boxes through decoding. With confidence level p, its output form for:

[0050] ;

[0051] in, For the i-th candidate box, Its confidence level, This represents the number of candidate boxes.

[0052] Step 4: Employ an event-driven online target tracking model. Based on the target's historical state, event time sequence distribution, and motion pattern, perform real-time correlation and position updates on detected targets to achieve stable tracking output.

[0053] In step 4, the event-driven tracking module is used to achieve real-time association and status updates of the detected target. This module adopts an integrated detection-tracking structure, consisting of a Kalman prediction unit and an event flow association unit.

[0054] First, based on the target state at the previous moment (Including position, velocity, and scale) Perform Kalman prediction to obtain the predicted position of the current target. :

[0055] ;

[0056] in, Here is the state transition matrix. This represents system noise.

[0057] Subsequently, the detection results are matched with the predicted bounding boxes based on the event trigger strength. The joint association strategy with geometric overlap (IoU, Intersection over Union) is used to calculate the matching cost:

[0058] ;

[0059] in, and These are the weighting coefficients. The Hungarian algorithm is used to achieve globally optimal matching, yielding the final association result.

[0060] For targets that match successfully, perform a status update:

[0061] ;

[0062] in, For Kalman gain, For input detection.

[0063] Step 5: Fuse the detection and tracking results, and perform confidence correction on the results based on the event trigger intensity and target motion continuity to obtain the final UAV detection and tracking trajectory.

[0064] In step 5, after both target detection and tracking are completed, the results of the two are further fused to improve the stability and reliability of the final trajectory output.

[0065] First, the detection score With tracking prediction reliability A time series consistency assessment is performed, and the final confidence level is obtained through the following fusion strategy:

[0066] ;

[0067] in, The model is dynamically adjusted based on event density and background interference, making it more reliant on tracking in weak event segments and more reliant on detection in strong signal segments.

[0068] Based on this, the final UAV trajectory output is generated for targets that meet the confidence threshold:

[0069] .

[0070] Example:

[0071] All programs are based on the Python language and implemented in the PyTorch framework, using a single NVIDIA GeForce RTX3050 GPU for training and testing.

[0072] Compared to existing detection and tracking methods based on traditional frame images, this invention fully utilizes the data characteristics of event-stream vision. Through fixed-time-window event frame construction, spatiotemporal voxelization encoding, and a multi-scale lightweight feature extraction network, it achieves more stable feature modeling for weak UAV targets. Simultaneously, this invention deeply integrates detection and tracking, constructing an event-driven joint inference framework. This framework enables real-time, robust cross-frame correlation updates based on the continuity of event triggering and target motion patterns, effectively suppressing background interference and spatial noise. In complex scenarios (sky clouds, forest boundaries, extreme lighting changes, etc.), the method of this invention significantly outperforms existing mainstream methods in key metrics such as detection rate and false alarm rate. Comparisons with other methods are shown in Table 1.

[0073] Table 1

[0074]

[0075] It should be emphasized that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for UAV target detection and tracking based on event flow vision, characterized in that, Includes the following steps: Step 1: Obtain the raw event stream data output by the neuromorphic sensor, and use a fixed time window method to continuously segment the events to construct event frames; Step 2: Perform polarity filtering and spatial and temporal voxel encoding on the constructed event frames to extract key change information from the event data and form a salient feature representation of the weak UAV target; Step 3: A lightweight detection model based on the visual characteristics of events is used to detect the drone target regions that may exist in the salient feature representation, and to obtain the position and confidence output of the candidate targets. Step 4: Employ an event-driven online target tracking model to perform real-time correlation and position updates on detected candidate targets based on the target's historical state, event temporal distribution, and motion patterns, thereby achieving stable tracking output. Step 5: Fuse the detection and tracking results, and perform confidence correction on the results based on the event trigger intensity and target motion continuity to obtain the final UAV detection and tracking trajectory.

2. The UAV target detection and tracking method based on event flow vision according to claim 1, characterized in that: In step 1, the original event stream is continuously segmented using a fixed time window method, and event frames are constructed using a preset 33ms time window. During the frame construction process, noisy events are removed based on the consistency of event timestamps and the event density threshold to improve the usability of event frames in complex backgrounds.

3. The UAV target detection and tracking method based on event flow vision according to claim 1, characterized in that: In step 2, by performing polarity filtering, spatial neighborhood aggregation and temporal voxelization encoding on the event frames, the temporal distribution, polarity distribution and local event density of the events are mapped into multi-channel pseudo-image features to enhance the saliency representation of weak UAV targets.

4. The UAV target detection and tracking method based on event flow vision according to claim 1, characterized in that: In step 3, a lightweight detection model adapted to the visual characteristics of events is adopted. The model’s feature extraction capability in sparse event scenarios is improved by deformable convolutional units and multi-scale feature fusion structures. The position and confidence of candidate targets are generated by decoupled detection head structure.

5. The UAV target detection and tracking method based on event flow vision according to claim 1, characterized in that: In step 4, an event-driven online target tracking model is adopted. Based on historical detection boxes, the temporal continuity of event streams, and the target motion direction, an association strategy is constructed to update the detection results in real time, thereby achieving stable tracking of small UAVs in high-speed, weak event periods and occlusion scenarios.

6. The UAV target detection and tracking method based on event flow vision according to claim 1, characterized in that: In step 5, the timing consistency between the detection results and the tracking prediction is verified, and the confidence of the results is fused by combining the event triggering intensity and the continuity of target movement, thereby generating the final UAV target detection and tracking trajectory output.

7. The UAV target detection and tracking method based on event flow vision according to claim 2, characterized in that: In step 1, the events output by the neuromorphic sensor By quadruple composition: ； in, Represents pixel coordinates, For the event Occurrence timestamp, To represent the polarity of brightness changes, a fixed time window is used. The event stream data is segmented, and events within a continuous time window are aggregated into a single event frame. : ； at the same time, For window indexing, a joint filtering strategy of event density threshold and timestamp consistency is further adopted.

8. The UAV target detection and tracking method based on event flow vision according to claim 7, characterized in that: A joint filtering strategy based on event density threshold and timestamp consistency is adopted, including: for each newly arrived event... In the Centered on, with a spatial radius of Historical events are searched within the neighborhood of each pixel, and a time surface matrix is ​​introduced to record the timestamp of the most recent event for each pixel; a time consistency function is defined: ； in, The minimum time difference between the current event and all neighboring events. For the event The set of neighbor events, Timestamps of previous events within the neighborhood. The preset time-related threshold is used; if the time difference between the current event and its nearest neighboring event is less than... If so, the event is considered to be a related event caused by the target's motion, rather than random noise.

9. The UAV target detection and tracking method based on event flow vision according to claim 8, characterized in that: within a fixed time window Within this context, the distribution density of events that satisfy the time consistency check in the local space is statistically analyzed; a judgment criterion is defined: if the time correlation threshold is met... Within a local neighborhood, the total number of associated events exceeds the density threshold. If the event is positive, then retain it; otherwise, classify it as an isolated noise event and discard it.

10. The UAV target detection and tracking method based on event flow vision according to claim 7, characterized in that: In step 2, for each event frame First, perform channel-specific statistics based on event polarity: ； ； Subsequently, the time dimension is divided into B equally spaced sub-intervals, a three-dimensional voxel grid is constructed, and the events are spatiotemporally encoded: ； in, This represents the voxel intensity value. For the b-th time interval, For indicator functions; local density normalization and spatial neighborhood aggregation operations are introduced after voxel encoding: ； in, This is the output after normalization. It represents the pixel neighborhood region.

Images