A method and system for detecting and tracking unmanned aerial vehicles
Patent Information
- Application Number
- CN202611207065.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-10-30
Smart Images
Figure CN123015953A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, specifically to a detection and tracking method and system for unmanned aerial vehicles (UAVs). Background Technology
[0002] In recent years, the low-altitude economy and drone technology have developed rapidly, with low-cost, miniaturized, and easy-to-control drones being widely used in military reconnaissance, public safety, and logistics transportation. Low-altitude, slow-moving, and small drone targets typically exhibit characteristics such as small target scale, low contrast, and indistinct texture features in visible light images. Coupled with factors such as complex background interference, high-speed maneuvering, occlusion, and short-term disappearance, existing general-purpose algorithms face severe challenges in this scenario.
[0003] Existing UAV detection technologies mainly include: radar detection, which has limited accuracy for detecting low, slow, and small targets, and is costly and difficult to deploy; radio frequency detection, which relies on communication links and is not adaptable to autonomously flying or silent targets; and infrared / acoustic detection, which is easily limited by environmental noise, distance, and resolution.
[0004] Meanwhile, existing technologies still suffer from the following technical shortcomings: insufficient receptive field; standard convolution expands the receptive field with a quadratic increase in parameters, making it difficult to achieve sufficient context modeling capabilities under lightweight constraints, and failing to effectively distinguish drones from similar interfering objects such as birds and kites; loss of high-frequency information; traditional max pooling or stride convolution directly discards high-frequency subband information during downsampling, while discriminative details such as the edges and rotors of drone targets are concentrated in high-frequency components, resulting in low recall rates for small targets; failure of multi-scale fusion; existing detection heads process multi-scale features with fixed weights, and on datasets where the target scale distribution is severely biased towards small targets, the network cannot adaptively amplify the contribution of high-resolution shallow features; lack of geometric constraints; standard IoU only measures the overlapping area and cannot distinguish between "reasonable offset along the direction of motion" and "erroneous drift in the vertical direction," easily leading to lateral false associations in drone maneuvering scenarios; standard IoU is insensitive to aspect ratio changes; when drones perform dives or climbs, the aspect ratio of the target projection can change by more than 40%, at which point the penalty for shape differences by IoU is insufficient. Summary of the Invention
[0005] This invention provides a method and system for detecting and tracking unmanned aerial vehicles (UAVs).
[0006] The technical solution of this invention is as follows: A method for detecting and tracking unmanned aerial vehicles (UAVs) includes the following steps: S1. Acquire the video frame to be detected and extract shallow features. Then, perform multiple sets of structure reparameterization and parameterless downsampling alternating processing to extract features. During this process, the first intermediate feature is obtained after the second structure reparameterization. After the last structure reparameterization, multi-scale receptive field enhancement and self-attention processing are performed to obtain deep features. After upsampling, these deep features are concatenated with the deep features and then subjected to structure reparameterization to obtain the second intermediate feature. After upsampling, these features are fused with the first intermediate feature and then subjected to structure reparameterization to obtain the third intermediate feature. After downsampling, these features are fused with the second intermediate feature and then subjected to structure reparameterization to obtain the fourth intermediate feature. After downsampling, these features are fused with the deep features and then subjected to structure reparameterization to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are normalized and weighted and fused to obtain the detection features. The detection features are decoded and classified to obtain the output drone detection result confidence score. S2. The confidence scores of the UAV detection results are divided into high-scoring detection boxes and low-scoring detection boxes. The compensated coordinates are obtained through camera motion compensation. Combined with the optimal estimate of the previous frame, the predicted UAV trajectory is obtained. The UAV trajectory is first associated with the high-scoring detection boxes using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are second associated with the low-scoring detection boxes based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV trajectory.
[0007] Furthermore, the structure is reparameterized, as follows: One path uses standard convolution, while the other path performs dilated convolution in parallel. The two outputs are concatenated and then fused by convolution. The dilated convolution reuses the weight tensor of the standard convolution, changing only the sampling coordinates, so the overall complexity remains unchanged.
[0008] Furthermore, parameterless downsampling decomposes the input signal into sub-signals of multiple scales, including low-frequency and high-frequency components.
[0009] Furthermore, the third, fourth, and fifth intermediate features are normalized and weighted and fused to obtain the detection features. The context-sensitive attention processing is shown below: Channel attention and spatial attention are concatenated on the fused features. Channel attention generates channel descriptors through global average pooling, and then maps them to importance weights for each channel through two fully connected layers. Spatial attention performs average pooling and max pooling along the channel dimension on the channel-calibrated features, and then generates a spatial weight map through convolution.
[0010] Furthermore, spatial matching loss is used to associate the detection box with the prediction box, i.e. the predicted trajectory, through a four-dimensional weighted fusion of angle loss, distance loss, shape loss and IoU loss.
[0011] Furthermore, in the first association matching, a joint cost matrix that integrates spatial matching loss and re-identified appearance feature distance is used to match high-resolution detection boxes with predicted trajectories using the Hungarian algorithm.
[0012] Furthermore, in the second association matching, for trajectories that were not successfully matched in the first association, the continuity of target motion is measured using low-scoring detection boxes based solely on spatial matching loss.
[0013] Furthermore, structural reparameterization is performed, particularly for height... The amount of contextual information captured by a wide drone target is determined as follows: in, For contextual information; These are pixel coordinates; For position Weighting coefficients at each location; For position Eigenvalues at; To expand the neighborhood covered by the convolution; The center pixel coordinates represent the point currently being processed. This represents the basic size parameter of the convolution kernel; The expansion factor of the neighborhood is controlled by the expansion rate.
[0014] A detection and tracking system for unmanned aerial vehicles (UAVs) includes the following: The reparameterization detection module acquires the video frames to be detected and extracts shallow features. These shallow features are then processed through multiple sets of alternating structural reparameterization and parameterless downsampling. During this process, a second structural reparameterization process yields the first intermediate feature. After a final structural reparameterization process, multi-scale receptive field enhancement and self-attention processing are applied to obtain deep features. These deep features are then upsampled and concatenated with the deep features, followed by structural reparameterization to obtain the second intermediate feature. This second intermediate feature is then upsampled, fused with the first intermediate feature, and reparameterized again to obtain the third intermediate feature. This third intermediate feature is downsampled, fused with the second intermediate feature, and reparameterized again to obtain the fourth intermediate feature. Finally, this fifth intermediate feature is downsampled, fused with the deep features, and reparameterized again to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are then normalized and weighted to obtain the detection features. Finally, the detection features are decoded and classified to obtain the output drone detection result confidence score. The hierarchical association tracking module divides the confidence of the UAV detection results into high-scoring detection boxes and low-scoring detection boxes. It obtains the compensated coordinates through camera motion compensation and combines them with the best estimate of the previous frame to predict the UAV's motion trajectory. The UAV trajectory is then matched with the high-scoring detection box using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are matched with the low-scoring detection boxes a second time based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV motion trajectory.
[0015] Furthermore, the reparameterization detection module includes a structure reparameterization processing module, one of which performs standard convolution and the other performs dilated convolution in parallel; the two outputs are convolved and then fused, and the dilated convolution reuses the weight tensor of the standard convolution, only changing the sampling coordinates, so the overall complexity remains unchanged.
[0016] The beneficial effects of this invention are as follows: 1. This invention employs a dilated-residual dual-branch complementary architecture. The dilated branch captures large-scale semantic context, while the residual branch preserves high-resolution spatial details through identity mapping. The two are concatenated to achieve a decoupled representation of details and semantics. Strictly adhering to the principle of zero parameter growth—the dilated convolution reuses the standard convolution weight tensor, changing only the sampling coordinates, while maintaining the overall model complexity. This effectively alleviates the classic trade-off between details and semantics in small object detection, significantly suppressing false alarms caused by similar interfering objects such as birds and kites and reducing the false negative rate.
[0017] 2. This invention employs a parameterless downsampling mechanism based on discrete wavelet transform, which reduces the resolution by a factor of 2 while ensuring energy conservation and fully preserving the discriminative details of high-frequency subbands such as edges and rotors. This effectively improves the recall rate of tiny UAV targets with pixel areas less than 0.1% of the total image area, while reducing the number of parameters by 26.4%, thus balancing performance and efficiency.
[0018] 3. This invention performs normalized weighted fusion of three layers of features, adaptively enhancing the proportion of shallow high-resolution features. Simultaneously, it concatenates channels and spatial attention to suppress redundant background channels, effectively reducing the false detection rate in complex backgrounds. For the loss mechanism, Focal Loss and CIoU Loss are jointly optimized; the former focuses on difficult samples such as small targets to improve recall, while the latter alleviates the problem of unstable bounding box regression gradients through geometric constraints.
[0019] 4. This invention achieves precise association between detection boxes and prediction boxes through weighted fusion of angle loss, distance loss, shape loss, and IoU loss. Angle loss penalizes the deviation between the direction of the line connecting the center points and the main direction of the prediction box, providing directional constraints for two typical maneuvers: horizontal cruise and vertical climb. Distance loss employs a dynamic weighting mechanism linked to angle loss; the greater the directional deviation, the stronger the distance penalty. When the angle is aligned, positional accuracy is enhanced, avoiding false link breaks caused by rigid thresholds. Shape loss penalizes aspect ratio differences, mitigating deformation problems caused by changes in UAV pitch angle and preventing normal maneuvers from being misjudged as trajectory changes. The weights of IoU, angle, distance, and shape loss in the overall cost reflect a priority design of "overlap first, direction second, distance third, and shape as a fallback," with negative values serving as hard rejection signals.
[0020] 5. This invention expands the receptive field and enhances discriminative ability through structural reparameterization, preserves edge details through parameterless downsampling, and adaptively fuses multi-scale features using large-small context sampling and boundary discrimination detection heads to output high-precision detection boxes. The tracker uses spatial matching loss with four-dimensional constraints for soft-decision association. During occlusion, angle-distance linkage allows for a relaxation of the distance threshold to maintain trajectory continuity. When the UAV performs high-speed maneuvers, the flexible constraints of shape loss prevent reasonable deformation from being misjudged as trajectory breakage, reducing ID switching rate and improving tracking stability. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the hardware integration of the present invention; Figure 2 This is a coastal background experimental diagram for the present invention. Detailed Implementation
[0022] To better understand the above technical solutions, a detailed description of the solutions will be provided below in conjunction with the accompanying drawings and specific embodiments. It should also be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of the invention.
[0023] This embodiment provides a method for detecting and tracking unmanned aerial vehicles (UAVs), including the following steps: S1. Acquire the video frames to be detected and extract shallow features. These shallow features are then processed through multiple sets of alternating structural reparameterization and wavelet pooling. During this process, a second structural reparameterization process yields the first intermediate feature. After the final structural reparameterization process, multi-scale receptive field enhancement and self-attention processing are applied to obtain deep features. These deep features are then upsampled and concatenated with the deep features, followed by structural reparameterization to obtain the second intermediate feature. After upsampling and fusion with the first intermediate feature, structural reparameterization is applied again to obtain the third intermediate feature. After downsampling and fusion with the second intermediate feature, structural reparameterization is applied again to obtain the fourth intermediate feature. After downsampling and fusion with the deep features, structural reparameterization is applied again to obtain the fifth intermediate feature. Normalized weighted fusion of the third, fourth, and fifth intermediate features yields the detection features. The detection features are then decoded and classified to obtain the output detection result confidence score.
[0024] In this embodiment of the invention, in view of the specific scale characteristics of low, slow and small UAV targets, and further, for the detection scenario of low, slow and small UAVs, a specific combination and collaborative workflow of three technologies are designed, namely, structural reparameterization processing, parameterless downsampling processing, and LSCSBD detection head (large-small context sampling and boundary discrimination detection head), forming a full-link optimization from feature extraction, downsampling to detection output.
[0025] The system acquires drone video frames and performs feature extraction to obtain shallow features. These shallow features are then processed through multiple sets of alternating structural reparameterization and parameterless downsampling. During this process, a second structural reparameterization process yields the first intermediate feature. After a final structural reparameterization process, multi-scale receptive field enhancement and self-attention processing are performed to obtain deep features. These deep features are then concatenated with the upsampled features and subjected to structural reparameterization to obtain the second intermediate feature. After upsampling, these features are fused with the first intermediate feature and subjected to structural reparameterization to obtain the third intermediate feature. After downsampling, these features are fused with the second intermediate feature and subjected to structural reparameterization to obtain the fourth intermediate feature. Finally, after downsampling and fusion with the deep features, these features are subjected to structural reparameterization to obtain the fifth intermediate feature.
[0026] To overcome the problems of insufficient feature extraction capability and insufficient receptive field for long-range targets in the detection of low-speed, small UAVs, this invention introduces structural reparameterization processing. Furthermore, the structural reparameterization processing adopts an expanded-residual dual-branch complementary architecture, as shown below: One path uses a standard 3×3 convolution to preserve local details, while the other path performs a parallel 3×3 dilated convolution with a dilation rate of r=2 to capture the global context. The two outputs are concatenated and then fused by a 1×1 convolution, achieving an exponential expansion of the receptive field. At the convolutional level, structural reparameterization increases the receptive field from 15×15 to 27×27 pixels, a relative increase of 44%. Strict adherence to the zero-parameter growth principle is maintained: the dilated convolution reuses the weight tensor of the standard convolution, changing only the sampling coordinates, keeping the overall complexity unchanged. For a size of... Let the drone targets be defined by their height and width, respectively. The amount of contextual information captured by the structure reparameterization process can be modeled as follows: in, For contextual information; These are pixel coordinates; For position Weighting coefficients at each location; For position Eigenvalues at; To expand the neighborhood covered by convolution, false alarms caused by similar objects such as birds and kites are significantly suppressed; The center pixel coordinates represent the point currently being processed. This represents the basic size parameter of the convolution kernel; The expansion factor of the neighborhood is controlled by the expansion rate.
[0027] Specifically, the two branches complement each other: the residual branch preserves high-resolution spatial details through identity mapping, which is crucial for sub-pixel-level localization; the expansion branch captures large-scale semantic context.
[0028] Details of the concatenation and fusion operation implementation – semantic decoupling representation, alleviating the classic trade-off problem in small object detection. The derivative of the gradient flowing through the residual path is shown below: in, The loss function; For input features; For output features; For residual branches The convolution operation.
[0029] In this embodiment of the invention, to address the inherent limitations of standard convolutional downsampling in extracting features from small targets, a parameter-free downsampling mechanism based on discrete wavelet transform, WaveletPool, is designed. Wavelet transform converts the input signal... The signal is decomposed into sub-signals of multiple scales, typically including low-frequency and high-frequency components. This process can be expressed by the following formula: in, For wavelet transform coefficients, represents the weighting coefficients for different scales; In the first Wavelet transform results at each scale This is the final output. Through weighted summation, WaveletPool can preserve multi-scale information of the image while achieving downsampling and feature preservation. Compared to a single learnable filter with a convolutional layer of stride 2 (ConvStride2), wavelet decomposition reduces the resolution by a factor of 2 without losing any frequency information, and energy conservation ensures the complete transmission of feature information. For drone targets with a pixel area less than 0.1% of the total image area, WaveletPool effectively improves the recall rate of small targets by preserving discriminative details such as edges and rotors in the high-frequency subband.
[0030] The detection features are obtained by normalizing and weighting the third, fourth, and fifth intermediate features.
[0031] In this embodiment of the invention, to address the problems of the original detection head processing multi-scale features with fixed weights, lacking context awareness, and not optimizing for imbalanced small target samples, the present invention designs an LSCSBD detection head. The LSCSBD detection head introduces learnable scale weights in the style of BiFPN (Bidirectional Feature Pyramid Network) to normalize and weight-fuse the three layers of features. Each layer of features is first aligned to a uniform number of channels via a 1×1 convolution. Let the aligned features be... , , Indicates the first Feature layer. Weighted fusion is defined as: in, Indicates the first Layer features; These are learnable scalar weights; Constrain it to be non-negative; This indicates that the features of each layer will be upsampled or downsampled to the target resolution; It is a numerically stable term; This represents the features after weighted fusion.
[0032] This strategy enables the method of this invention to automatically learn the optimal scale importance ratio during training—for scenarios with dense small targets, such as low-speed, small drones, the convolutional layer weights... It will adaptively increase with training, so that shallow high-resolution features will account for a larger proportion in the fusion result, preserving the detailed information of the target edges and contours.
[0033] Furthermore, context-sensitive attention processing involves concatenating channel attention and spatial attention on the fused features. Channel attention generates channel descriptors through global average pooling, which are then mapped to importance weights for each channel via two fully connected layers. This suppresses redundant channels that strongly respond to background textures and activates effective channels that are sensitive to target edges and rotor frequency. Spatial attention performs average pooling and max pooling along the channel dimensions on the channel-calibrated features, followed by 7×7 convolution to generate a spatial weight map. This guides the detection head to concentrate computational resources on areas where the target may appear, effectively reducing the false detection rate even under interference from complex backgrounds such as buildings, clouds, and flocks of birds.
[0034] The detection features are decoded and classified to obtain the confidence level of the output UAV detection results.
[0035] In this embodiment of the invention, Focal Loss and CIoU Loss are jointly optimized: Addressing the issues of extremely small target pixel area and severe imbalance in the number of positive and negative samples in the training samples (the proportion of positive samples is typically less than 0.1%), Focal Loss is used to dynamically reduce the loss weight of easily classified negative samples. in, These are the category weight coefficients; This represents the model's predicted probability of the true class. This is the logarithmic term in the cross-entropy loss; To focus on controlling the degree of suppression of easily classified samples, hyperparameters are commonly used. Focal Loss through The factor automatically shifts the training focus to challenging samples such as small targets and low-confidence samples, effectively improving recall without changing the positive-to-negative sample ratio. Simultaneously, CIoU Loss is used instead of standard IoU Loss, incorporating geometric information into the optimization objective through center distance and aspect ratio constraints, thus mitigating the problem of unstable regression gradients for small target bounding boxes.
[0036] In this invention, structural reparameterization expands the receptive field and enhances contextual modeling capabilities, laying the foundation for subsequent feature extraction. WaveletPool preserves high-frequency details during the downsampling stage, ensuring that discriminative features such as rotors are not lost. The LSCSBD detection head achieves adaptive fusion of multi-scale features and attention enhancement during the output stage, and optimizes loss for imbalanced small target samples. These three components form a complete technical chain of "feature extraction - feature preservation - feature fusion," working synergistically throughout the entire process of detecting low-speed, small UAVs.
[0037] S2. The confidence scores of the UAV detection results are divided into high-scoring detection boxes and low-scoring detection boxes. The compensated coordinates are obtained through camera motion compensation. Combined with the optimal estimate of the previous frame, the predicted UAV trajectory is obtained. The UAV trajectory is first associated with the high-scoring detection boxes using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are second associated with the low-scoring detection boxes based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV trajectory.
[0038] In this embodiment of the invention, the confidence level of the UAV detection result is divided into high-resolution detection boxes and low-resolution detection boxes. First, camera motion compensation is used to estimate the global motion changes between consecutive video frames, and the target state predicted by the Kalman filter is corrected to reduce the impact of camera motion on the target trajectory prediction. Based on this, an improved Kalman filter is used to predict the target's position state at the next moment, and the matching cost between the predicted trajectory and the detection result is calculated using the Hungarian algorithm, realizing data association between the detected target and historical trajectories. After a successful match, the current detection result is used as the observation input to the Kalman filter to recursively update the target state, ultimately forming a continuous and stable target tracking trajectory.
[0039] Specifically, the system first uses Kalman filtering combined with camera motion compensation to obtain predicted trajectories. In the first association matching stage, a joint cost matrix fusing SIoU (Spatial Matching Loss) and ReID (Re-identification) appearance feature distances is used to achieve accurate matching between high-resolution detection boxes and predicted trajectories via the Hungarian algorithm. For trajectories that fail to match in the first association, a second association matching is performed using low-resolution detection boxes, relying solely on SIoU to measure the continuity of target motion, thereby recovering trajectories lost due to occlusion, scale changes, or decreased detection confidence. For successfully matched trajectories, the current detection result is used as observation information input to the Kalman filter, and a more accurate target motion state is obtained through state updates. In other words, the second association matching is a supplementary process after the first association matching failure, used to improve the recall rate of low-confidence targets and avoid ID loss due to decreased detection confidence. Furthermore, for unmatched targets that meet the initialization conditions, new trajectories are established and assigned corresponding IDs; for trajectories that fail to match for multiple consecutive frames, they are retained or deleted based on their lifecycle state. Finally, trajectory updates, new trajectory creation, and invalid trajectory deletion are completed based on the two-stage matching results.
[0040] In this embodiment of the invention, improved detection accuracy lays the foundation for tracking performance. However, the standard IoU metric still suffers from problems such as excessive displacement sensitivity, lack of directional and geometric constraints, and insufficient scale adaptability in tracking slow, small UAVs. An improved S-Bot-SORT algorithm is proposed, using SIoU instead of standard IoU as the motion similarity metric. Through a four-dimensional weighted fusion of angle loss, distance loss, shape loss, and IoU loss, a soft-decision association between the detection box and the predicted box (i.e., the predicted trajectory) is achieved. The specific process is as follows: Angle loss: By penalizing the deviation between the direction of the line connecting the center points and the main direction of the predicted box, the predicted box is guided to align with the direction of the true box.
[0041] Define the true detection box With prediction box The angle between the line connecting the center points and the horizontal axis is The angle between the vertical axis and the vertical axis is ; in, Representing the One real detection frame; Representing the One prediction box; For the true frame The coordinates of the center point, Represents the x-coordinate of a pixel. Represents the ordinate of a pixel; For the true frame Height and width; For prediction boxes The coordinates of the center point; For prediction boxes Height and width.
[0042] The angle loss component is designed as a piecewise function: Among them, intermediate variables , The formula for calculation is: The minimum rectangular width to cover the two frames. When the line connecting the centers of the two frames is close to horizontal, horizontal deviation is penalized first; when it is close to vertical, vertical deviation is penalized first, providing directional constraints for the two typical maneuvering modes of the UAV: horizontal cruise and vertical climb.
[0043] Distance loss: In addition to the angle loss, a further penalty is imposed on the Euclidean distance of the center point. The distance loss components are defined as follows: in, Normalized center distance; This is a sensitivity coefficient linked to angle loss. When the angle deviation... When it approaches 0, Approaching 2, the distance penalty increases; when angles are aligned When it approaches 1, As the distance approaches 1, the distance penalty weakens, ensuring that the greater the directional deviation, the higher the distance accuracy requirement. When drone maneuvers cause increased angular deviation, the distance penalty weight is automatically reduced to avoid false chain breaks caused by rigid distance constraints; when the angle is aligned, the distance accuracy requirement is strengthened to ensure positional accuracy during stable tracking. This dynamic weighting mechanism alleviates the rigidity problem of fixed threshold matching in Bot-SORT under maneuvering scenarios.
[0044] Shape loss penalizes aspect ratio differences to address bounding box deformation caused by changes in UAV pitch angle. Shape loss components are defined as follows: in, The shape sensitivity index is set as follows, specifically for the rapid size-dependent characteristics of UAVs. To avoid excessive penalties during scaling, reasonable scaling should be avoided, and normal maneuvering deformation should not be misjudged as trajectory switching. To iterate over variables, take Width and Two dimensions, height For the true frame in dimension The value on, For the predicted bounding box in dimension The value on; It is an exponential function; To eliminate the scale effect on the relative difference rate.
[0045] Synthetic SIoU cost function: in, This is the actual detection frame. For predicting detection boxes. The denominator design reflects priority: IoU loss weight 1, angle loss... Weight is 0.5, distance loss Weight 0.25, shape loss Weight 0.125. SIoU range is... Negative values appear when IoU is extremely low and geometric deviation is extremely large, serving as a hard rejection signal.
[0046] The present invention's S1 and S2 form a feature-geometric dual-constraint closed loop: structural reparameterization processing expands the receptive field during feature extraction through dilatation-residual dual-branching, enhancing the ability to distinguish between drones and interfering objects such as birds and kites; parametric downsampling processing preserves multi-scale information of the image, enhancing the ability to extract edge information of drones; the LSCSBD detection head adaptively fuses multi-scale features through learnable scale weights to output high-precision detection boxes; based on this, the S-Bot-SORT tracker uses SIoU's four-dimensional geometric constraints to perform soft-decision association between detection boxes and motion prediction boxes. When the drone is briefly occluded in complex background clouds or buildings, the detection layer may output low-confidence boxes. The tracker, through SIoU's angle-distance linkage mechanism, allows for a relaxation of the distance threshold under the premise of consistent direction, maintaining trajectory continuity; when the drone performs high-speed maneuvers, the flexible constraint of shape loss avoids misjudging reasonable deformation as trajectory breakage.
[0047] This embodiment provides a detection and tracking system for unmanned aerial vehicles (UAVs), including the following: The reparameterization detection module acquires UAV video frames and extracts shallow features. These shallow features are then processed through multiple sets of alternating structural reparameterization and parameterless downsampling. During this process, a second structural reparameterization process yields the first intermediate feature. After a final structural reparameterization process, multi-scale receptive field enhancement and self-attention processing are applied to obtain deep features. These deep features are then concatenated with the upsampled features and subjected to structural reparameterization to obtain the second intermediate feature. After upsampling, this feature is fused with the first intermediate feature and subjected to structural reparameterization to obtain the third intermediate feature. After downsampling, this feature is fused with the second intermediate feature and subjected to structural reparameterization to obtain the fourth intermediate feature. After downsampling and fusion with deep features, this feature is subjected to structural reparameterization to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are then normalized and weighted to obtain the detection features. Finally, the detection features are decoded and classified to obtain the output UAV detection result confidence score. The hierarchical association tracking module divides the confidence of the UAV detection results into high-scoring detection boxes and low-scoring detection boxes. It obtains the compensated coordinates through camera motion compensation and combines them with the best estimate of the previous frame to predict the UAV's motion trajectory. The UAV trajectory is then matched with the high-scoring detection box using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are matched with the low-scoring detection boxes a second time based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV motion trajectory.
[0048] In this embodiment of the invention, a hardware and software co-working embedded detection and tracking system is designed. Based on the NVIDIA Jetson Xavier NX platform, it achieves real-time inference through a three-stage optimization process of "PyTorch→ONNX→TensorRT", and finally assigns the ID number of the tracked target to the video stream in real time and stably.
[0049] The essence of the three-stage process is to progressively reduce the level of abstraction and progressively deepen hardware adaptation: PyTorch provides algorithmic flexibility, ONNX achieves framework-independent standardization, and TensorRT completes extreme optimization for NVIDIA GPUs. Through layer fusion, quantization, kernel tuning, and asynchronous pipelines, inference latency can be compressed from tens of milliseconds native to PyTorch to several milliseconds or even sub-milliseconds, meeting the stringent latency requirements of scenarios such as real-time video analysis, autonomous driving, and industrial inspection.
[0050] Phase 1: Convert PyTorch's dynamic computation graph to ONNX. Use PyTorch's native `torch.onnx.export()` function to complete the export. First, prepare the model instance and sample input tensors, then call the export function. Key parameters include `input_names` and `output_names`, `dynamic_axes`, and `opset_version`. During export, the model should be set to `eval` mode, and it is recommended to perform a forward propagation first to ensure the weights are loaded.
[0051] Phase Two: The core phase of performance enhancement. TensorRT, as NVIDIA's inference optimizer, performs aggressive hardware-level optimizations to the computation graph. It uses trt.OnnxParser to directly parse ONNX files, construct network definitions, and then creates the optimized engine using IBuilder.
[0052] Phase 3: The optimized TensorRT engine needs to be deployed for low-latency inference in the target environment. This uses TensorRT's ICudaEngine and IExecutionContext interfaces. First, the engine file is deserialized to create the execution context, and then the GPU memory buffer is prepared. During inference, the preprocessed data is asynchronously transferred to the GPU via cuda.memcpy_htod_async, the inference kernel function is executed by calling context.execute_async_v3, and finally, the result is returned to the CPU via cuda.memcpy_dtoh_async.
[0053] Reference Appendix Figure 1The diagram shows the hardware integration. The NN-Q40XT50 photoelectric turntable transmits visible light video streams to the Jetson Xavier NX via Gigabit Ethernet using the RTSP protocol, meeting the requirements for low-latency real-time processing. The Jetson Xavier NX connects to an industrial monitor via HDMI 2.0 to output a visual interface, and connects to a keyboard and mouse via USB 3.0 for local control. USB and MIPI CSI interfaces are also reserved for expanding auxiliary sensors. The photoelectric turntable uses DC 24V / 5A power, while the Jetson Xavier NX uses DC 19V / 15W power, ensuring stability for outdoor deployment. This solution deeply integrates photoelectric sensing and AI computing capabilities, constructing a hardware closed loop from physical perception to decision execution, providing a reliable physical foundation for the engineering deployment of algorithms.
[0054] Specifically, to comprehensively evaluate the engineering practicality of the constructed system, a series of quantitative and qualitative tests were designed and executed in real outdoor scenarios. The tests aimed to verify the system's real-time performance, tracking accuracy, stability, and environmental adaptability. By combining quantitative indicators with visualization effects, the actual performance of the YOLO11-DWL detection algorithm and the S-Bot-SORT tracking algorithm on the embedded platform was objectively evaluated. Hardware-wise, a Jetson XavierNX, an NN-Q40XT50 turntable, and an industrial monitor were used. Software configuration included JetPack 4.5.1, CUDA 10.2, TensorRT 8.2, PyTorch 1.6.0, and Python 3.10. Regarding experimental constraints, a DJI Mavic AIR 2 was used as the target drone, with flight speed limited to no more than 50 m / s and flight range limited to no more than 100 m, thus achieving the flight characteristics of a low-altitude, slow, and small drone. Tests were conducted in coastal scenarios with strong lighting and complex, variable backgrounds. During system operation, the video stream is acquired via RTSP and preprocessed. The target detection is then performed by the method of this invention. After structural reparameterization for feature extraction, bounding boxes are output by the WaveletPool and LSCSBD detection heads. The detection results are input into the S-Bot-SORT tracker, and trajectory association and updates are achieved through Kalman filtering prediction, camera motion compensation, SIoU cost matrix construction, and Hungarian matching. Finally, the detection results are rendered and output in real time.
[0055] Reference Appendix Figure 2 The image shows an experimental view against a coastal background. The system can accurately identify small UAV targets at long distances, effectively distinguishing the target from distant, blurred-edge objects such as reefs and waves. The S-Bot-SORT tracker assigns a stable and unique ID to the target, maintaining tracking continuity even when the target is briefly obscured or undergoes drastic scale changes.
Claims
1. A method for detecting and tracking unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1. Acquire the video frame to be detected and extract shallow features. Then, perform multiple sets of structure reparameterization and parameterless downsampling alternating processing to extract features. During this process, the first intermediate feature is obtained after the second structure reparameterization. After the last structure reparameterization, multi-scale receptive field enhancement and self-attention processing are performed to obtain deep features. After upsampling, these deep features are concatenated with the deep features and then subjected to structure reparameterization to obtain the second intermediate feature. After upsampling, these features are fused with the first intermediate feature and then subjected to structure reparameterization to obtain the third intermediate feature. After downsampling, these features are fused with the second intermediate feature and then subjected to structure reparameterization to obtain the fourth intermediate feature. After downsampling, these features are fused with the deep features and then subjected to structure reparameterization to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are normalized and weighted and fused to obtain the detection features. The detection features are decoded and classified to obtain the output drone detection result confidence score. S2. The confidence scores of the UAV detection results are divided into high-scoring detection boxes and low-scoring detection boxes. The compensated coordinates are obtained through camera motion compensation. Combined with the optimal estimate of the previous frame, the predicted UAV trajectory is obtained. The UAV trajectory is first associated with the high-scoring detection boxes using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are second associated with the low-scoring detection boxes based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV trajectory.
2. The method for detecting and tracking a drone according to claim 1, characterized in that, The structural reparameterization process described in S1 is as follows: One path uses standard convolution, while the other path performs dilated convolution in parallel. The two outputs are concatenated and then fused by convolution. The dilated convolution reuses the weight tensor of the standard convolution, changing only the sampling coordinates, so the overall complexity remains unchanged.
3. The method for detecting and tracking a drone according to claim 1, characterized in that, The parameterless downsampling described in S1 decomposes the input signal into sub-signals of multiple scales, including low-frequency components and high-frequency components.
4. The method for detecting and tracking a drone according to claim 1, characterized in that, The detection features are obtained by normalizing and weighting the third, fourth, and fifth intermediate features as described in S1. The context-sensitive attention processing is as follows: Channel attention and spatial attention are linked together in the fusion feature. Channel attention generates channel descriptors through global average pooling, and then maps them to the importance weights of each channel through two fully connected layers. Spatial attention performs average pooling and max pooling along the channel dimension on the channel-calibrated features, and then generates a spatial weight map through convolution.
5. The method for detecting and tracking a drone according to claim 1, characterized in that, The spatial matching loss described in S2 associates the detection box with the prediction box, i.e. the prediction trajectory, through a four-dimensional weighted fusion of angle loss, distance loss, shape loss and IoU loss.
6. The method for detecting and tracking a drone according to claim 1, characterized in that, The first association matching described in S2 uses a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance, and performs matching between high-resolution detection boxes and predicted trajectories using the Hungarian algorithm.
7. The method for detecting and tracking a drone according to claim 1, characterized in that, The second association matching, for trajectories that were not successfully matched in the first association, uses low-scoring detection boxes to measure the continuity of target motion based solely on spatial matching loss.
8. The method for detecting and tracking a drone according to claim 1, characterized in that, The structural reparameterization process described in S1, for height The amount of contextual information captured by a wide drone target is determined as follows: in, For contextual information; These are pixel coordinates; For position Weighting coefficients at each location; For position Eigenvalues at; To expand the neighborhood covered by the convolution; The center pixel coordinates represent the point currently being processed. This represents the basic size parameter of the convolution kernel; The expansion factor of the neighborhood is controlled by the expansion rate.
9. A detection and tracking system for unmanned aerial vehicles (UAVs), used to implement the UAV detection and tracking method according to claim 1, characterized in that, Includes the following: The reparameterization detection module acquires the video frames to be detected and extracts shallow features. These shallow features are then processed through multiple sets of alternating structural reparameterization and parameterless downsampling. During this process, a second structural reparameterization process yields the first intermediate feature. After a final structural reparameterization process, multi-scale receptive field enhancement and self-attention processing are applied to obtain deep features. These deep features are then upsampled and concatenated with the deep features, followed by structural reparameterization to obtain the second intermediate feature. This second intermediate feature is then upsampled, fused with the first intermediate feature, and reparameterized again to obtain the third intermediate feature. This third intermediate feature is downsampled, fused with the second intermediate feature, and reparameterized again to obtain the fourth intermediate feature. Finally, this fifth intermediate feature is downsampled, fused with the deep features, and reparameterized again to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are then normalized and weighted to obtain the detection features. Finally, the detection features are decoded and classified to obtain the output drone detection result confidence score. The hierarchical association tracking module divides the confidence of the UAV detection results into high-scoring detection boxes and low-scoring detection boxes. It obtains the compensated coordinates through camera motion compensation and combines them with the best estimate of the previous frame to predict the UAV's motion trajectory. The UAV trajectory is then matched with the high-scoring detection box using a joint cost matrix that fuses spatial matching loss and re-identified appearance feature distance. The unmatched UAV trajectories are matched with the low-scoring detection boxes a second time based on spatial matching loss. All successfully matched UAV trajectories are used as the updated UAV motion trajectory.
10. A detection and tracking system for a drone according to claim 9, characterized in that, The reparameterization detection module includes a structure reparameterization processing module, one of which performs standard convolution and the other performs dilated convolution in parallel. The two outputs are concatenated and then fused by convolution. The dilated convolution reuses the weight tensor of the standard convolution, only changing the sampling coordinates, so the overall complexity remains unchanged.