An unmanned aerial vehicle target tracking method based on visual recognition

Abstract

The application discloses a kind of unmanned aerial vehicle target tracking methods based on visual identification, method includes: the image data of unmanned aerial vehicle tracking target is collected;YOLO v5n model is utilized to the intermittent target identification detection of image after light weight processing, and the identification detection result is output;Identification detection result and the tracking information of KCF tracker are fused, and specific target is tracked, according to the identification detection result of YOLO v5n model, the tracking state of KCF tracker is dynamically calibrated, while according to the tracking condition of KCF tracker, the identification detection frequency of YOLO v5n model is dynamically adjusted, and the motion state of target is predicted based on Kalman filter, and the motion information of target is output;According to the motion information of target, flight control instruction is generated, and unmanned aerial vehicle is driven to track target.The application continuously tracks in complex background, light occlusion, target variable speed motion scene, and the drift time is lengthened, the efficiency of reacquisition after target loss is improved, and the stability and continuity of dynamic target tracking are improved.

Description

Technical Field

[0001] This invention relates to the field of drone tracking technology, specifically a drone target tracking method based on visual recognition. Background Technology

[0002] With the rapid development of drone technology, drones are increasingly used in security monitoring, low-altitude reconnaissance, and collaborative operations, making the demand for automated tracking of specific targets (such as low-altitude, slow-moving, and small drones, vehicles, and personnel) increasingly urgent. Traditional target tracking technologies mostly rely on single tracking algorithms (such as Meanshift and CAMShift), which have significant limitations: First, they are easily affected by factors such as target occlusion and complex backgrounds, leading to tracking drift or even target loss; second, they lack the ability to identify specific target categories, failing to distinguish targets from interference objects (such as birds and balloons), resulting in a high false detection rate; and third, they have poor adaptability to the movement of dynamic targets, making it difficult to cope with complex motion states such as target speed changes and turns.

[0003] Meanwhile, the deep integration of visual recognition with UAV flight control faces many challenges: on the one hand, visual recognition has inherent latency, and if the recognition algorithm consumes a lot of computing power, it will lead to untimely updates of target position information, which in turn will cause tracking lag; on the other hand, the target position output by visual recognition is easily affected by image noise and jitter, and directly converting it into flight control commands will cause the UAV's flight attitude to oscillate and the image to shake, which will seriously affect the tracking stability; in addition, the computing power and memory of airborne embedded devices (such as Raspberry Pi) are limited, and traditional complex visual models are difficult to adapt, which further exacerbates the contradiction between "recognition accuracy, real-time performance and flight smoothness".

[0004] Existing technologies have failed to effectively solve the above problems, specifically in the following ways: some solutions only optimize a single module, such as focusing only on model lightweighting or only designing control algorithms, without forming a full-link collaboration; detection and tracking work independently or are only in a simple switching relationship, lacking a deep integration mechanism; model optimization does not take into account the characteristics of "low, slow and small" targets, resulting in insufficient detection accuracy for small targets; flight control linkage design ignores command smoothness and hardware adaptability, leading to defects such as poor tracking stability, high latency and weak environmental adaptability in practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a visual recognition-based method for tracking unmanned aerial vehicles (UAVs) targets, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a visual recognition-based unmanned aerial vehicle (UAV) target tracking method, the method comprising:

[0007] Collect image data of drones tracking targets;

[0008] The YOLO v5n model, after being lightened, is used to perform intermittent target recognition and detection on the image, and the recognition and detection results are output.

[0009] The identification and detection results are fused with the tracking information of the KCF tracker to track a specific target. The tracking state of the KCF tracker is dynamically calibrated based on the identification and detection results of the YOLO v5n model. At the same time, the identification and detection frequency of the YOLO v5n model is dynamically adjusted based on the tracking situation of the KCF tracker. The motion state of the target is predicted based on the Kalman filter, and the motion information of the target is output.

[0010] Flight control commands are generated based on the target's motion information to drive the drone to track the target.

[0011] Furthermore, the process of fusing the identification and detection results with the tracking information of the KCF tracker to track a specific target, and dynamically calibrating the tracking state of the KCF tracker based on the identification and detection results of the YOLO v5n model, includes:

[0012] The KCF tracker is initialized with the recognition results of the first full-image detection using the YOLO v5n model;

[0013] In the normal tracking phase, the target tracking position is output frame by frame only through the KCF tracker, while the target position data of the most recent preset number of frames is cached.

[0014] After a preset interval of image frames, the YOLO v5n model is triggered to perform full-image detection again. The tracking of the KCF tracker is corrected based on the recognition and detection results of the YOLO v5n model. After the correction is completed, the tracking returns to the normal tracking stage to continue tracking.

[0015] Furthermore, the correction of the KCF tracker's tracking based on the recognition and detection results of the YOLO v5n model includes:

[0016] Calculate the intersection-over-union (IoU) value between the detection bounding boxes of the YOLO v5n model and the tracking bounding boxes of the KCF tracker;

[0017] If the IoU value is greater than or equal to the first matching threshold, the tracking is determined to be stable, and the tracking results of the KCF tracker are calibrated using the detection results of the YOLO v5n model.

[0018] If the IoU value is less than the first matching threshold and greater than the second matching threshold, it is determined to be a slight drift. The tracking result of the KCF tracker is retained, and the YOLO v5n model is triggered to perform full-image detection in the next frame in advance.

[0019] If the IoU value is less than or equal to the second matching threshold, it is determined to be a severe drift or target switching, the current KCF tracker is terminated, and the KCF tracker is reinitialized using the detection results of the YOLO v5n model.

[0020] Furthermore, when dynamically calibrating the tracking state of the KCF tracker based on the recognition and detection results of the YOLO v5n model:

[0021] A circular buffer is established as a synchronization cache pool for the YOLO v5n model and the KCF tracker, storing the recognition and detection data of the YOLO v5n model and the tracking data of the KCF tracker for the most recent preset number of image frames;

[0022] Set a synchronization flag to indicate the data synchronization status, and perform multi-threaded data reading and writing through atomic operations;

[0023] The synchronization flag is updated based on the IoU value, and the latest data status in the cache pool is updated synchronously.

[0024] Furthermore, the dynamic adjustment of the YOLO v5n model's recognition and detection frequency based on the tracking performance of the KCF tracker includes:

[0025] Calculate the bounding box area of ​​the target currently being tracked by the KCF tracker;

[0026] If the bounding box area is greater than the first pixel threshold, then a first detection interval is set;

[0027] If the bounding box area is less than or equal to the first pixel threshold and greater than the second pixel threshold, then a second detection interval less than the first detection interval is set.

[0028] If the bounding box area is less than or equal to the second pixel threshold, then a third detection interval is set that is less than the second detection interval.

[0029] Furthermore, the lightweight YOLO v5n model is obtained through the following steps:

[0030] Collect multi-scene image datasets containing specific targets;

[0031] The K-Means clustering algorithm is used to perform cluster analysis on the true bounding boxes of targets in the image dataset, generating adaptive anchor box samples that match the target scale distribution.

[0032] The YOLO v5n model is trained using a combined loss function, the formula for which is as follows:

[0033]

[0034] in, Represents the CIoU loss function. Represents the Focal loss function; Represents the cross-entropy loss function. These correspond to the weights of the three loss functions, respectively. , The weighting ratio is 3:1:1;

[0035] Static quantization is performed on the trained YOLO v5n model to convert the non-quantization-sensitive layers in the YOLO v5n model to INT8 precision, while retaining FP16 precision for the quantization-sensitive layers, thus generating a lightweight YOLO v5n model.

[0036] Soft nonmaximum suppression (Soft-NMS) is used to post-process the predicted bounding boxes output by the lightweight YOLO v5n model inference, reducing missed detections in dense target scenes and retaining valid targets.

[0037] Furthermore, the calculation formula for the CIoU loss function is as follows:

[0038] in, This represents a balance factor used to adjust the degree of influence of the aspect ratio term. Indicates weight, Indicates the center coordinates of the prediction box ( ), Represents the center coordinates of the true bounding box , This represents the diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted and ground truth bounding boxes. This represents the squared Euclidean distance between the predicted bounding box and the ground truth bounding box;

[0039] The formula for calculating the Focal loss function is as follows:

[0040] in, This represents the positive and negative sample balance coefficient. This represents the hard sample mining coefficient. This represents the model's predicted probability for the target class. , Indicates an indicator function, These are the true labels of the samples. This represents the original probability that the target value output by the model is the positive class.

[0041] The formula for calculating the cross-entropy loss function is as follows:

[0042] in, These are the true labels of the samples. This represents the probability distribution predicted by the model, where n is the number of classification categories. An index representing the category.

[0043] Furthermore, the step of statically quantizing the trained YOLO v5n model, converting the non-quantization-sensitive layers in the YOLO v5n model to INT8 precision, and retaining FP16 precision for the quantization-sensitive layers to generate a lightweight YOLO v5n model includes:

[0044] Structural layer pruning was performed on the trained YOLO v5n model with FP16 accuracy to remove redundant channels in each convolutional layer.

[0045] Use PyTorch's torch.quantization tool to insert quantization / dequantization nodes into the pruned model;

[0046] The model with inserted quantization / dequantization nodes is calibrated using a calibration dataset, and the numerical distribution information of the input and output of each layer is collected.

[0047] Based on the calibration results, the layers that have the greatest impact on the model accuracy are labeled as quantization-sensitive layers. The FP16 accuracy of the quantization-sensitive layers is retained, while the non-sensitive layers are converted to INT8 accuracy, thus generating a static lightweight model.

[0048] Furthermore, the prediction of the target's motion state based on the Kalman filter, and the output of the target's motion information, include:

[0049] Based on the bounding box information of the tracked target output by the KCF tracker, the center coordinates and bounding box area of ​​the tracked target are extracted;

[0050] The change in the center coordinates and the rate of change in the bounding box area of ​​the tracked target are calculated using the sliding window method.

[0051] A target motion model is established based on a Kalman filter to predict the target's position in the next 1-2 frames and output the target's motion information. The state equation of the Kalman filter is: The observation equation is ,in, Indicates the discrete time step or image frame number. Represents the state vector. ,in, This indicates the center coordinates of the target in the image coordinate system. Indicates the target is The speed of movement in the direction, This represents the transpose operation of a vector, where A is the state transition matrix, B is the control input matrix, and B=0. Represents the control input vector. The process noise of the Kalman filter follows a Gaussian distribution. , Indicates a normal distribution. The variance is set as H represents the observation matrix, used to map the state vector to the observation space. The observation noise is represented by a Gaussian distribution. , Indicates a normal distribution. The variance is set as .

[0052] Furthermore, the step of generating flight control commands based on the target's motion information to drive the UAV to track the target includes:

[0053] The target position and velocity information in the image coordinate system are combined with the drone altitude and camera parameters to transform the target relative position in the drone body coordinate system.

[0054] The deviation of the target relative position after conversion is input into the position loop of the cascade PID controller to generate a speed command;

[0055] The speed command is filtered, and the target speed predicted by the Kalman filter is used as a feedforward quantity for superposition compensation.

[0056] Furthermore, the speed command is filtered, and the target speed predicted by the Kalman filter is used as a feedforward quantity for superposition compensation, including:

[0057] The target position information is smoothed by applying a moving average filter.

[0058] The speed command generated by the cascaded PID controller is low-pass filtered to filter out high-frequency noise.

[0059] Set the maximum acceleration and rate of change thresholds for the drone to limit the abrupt changes in flight control commands;

[0060] The target velocity predicted by the Kalman filter is used as a feedforward quantity and added to the velocity command to compensate for the target motion in advance.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] 1. This invention utilizes a deep fusion mechanism of "lightweight YOLO v5n model recognition and detection + KCF tracking + Kalman prediction" combined with periodic calibration and state synchronization strategies to effectively suppress tracking drift caused by factors such as occlusion and deformation. In complex backgrounds, mild occlusion, and target variable speed motion scenarios, the tracking success rate is significantly improved, the continuous tracking drift-free time is extended, and the efficiency of re-acquiring the target after loss is improved, thereby enhancing the stability and continuity of dynamic target tracking.

[0063] 2. By optimizing the YOLOv5n model from data, loss function, K-Means adaptive anchor boxes to post-processing, the recognition accuracy of small-scale, low-texture targets is significantly improved. The recognition capability for specific targets such as "low, slow, and small" drones and personnel is significantly enhanced, and the false detection rate and false negative rate are greatly reduced. It can effectively distinguish targets from interference objects. It adopts mixed-precision static quantization, which is suitable for low-computing-power airborne devices. The visual recognition speed is ≥10 FPS and the tracking speed is ≥15 FPS. Combined with standardized data interface (single frame data size ≤32 bytes, transmission delay ≤5ms) and direct communication with UART, the end-to-end latency from image acquisition to flight control response is ≤80ms. While ensuring high recognition accuracy for "low, slow, and small" targets, memory and CPU usage are controlled within the hardware constraints of low-computing-power airborne devices such as Raspberry Pi Zero 2W. No independent GPU is required, power consumption is low, and it is suitable for long-term drone operation.

[0064] 3. A dynamic detection interval mechanism based on target scale was established. When the target is close (large frame area), the detection frequency is reduced to save computing power; when the target is far away (small frame area) or the state is unstable, the detection frequency is increased to ensure accuracy, thus achieving an intelligent balance between computing power consumption and tracking performance.

[0065] 4. The control strategy of "two-stage filtering + feedforward compensation based on Kalman prediction" effectively filters out visual noise, compensates for system delay, limits command abrupt changes, makes the UAV tracking flight attitude stable, reduces the amplitude of UAV flight attitude oscillation by 40%, significantly reduces tracking image jitter, and makes speed and acceleration changes stable, thus improving the tracking experience. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the steps of the visual recognition-based UAV target tracking method of the present invention.

[0067] Figure 2 This is a flowchart of the UAV target tracking method based on visual recognition according to the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] like Figures 1-2 As shown, this embodiment provides a visual recognition-based drone target tracking method, which includes the following steps:

[0070] S1: Acquire image data of the target being tracked by the drone;

[0071] S2: Use the lightweight YOLO v5n model to perform intermittent target recognition and detection on the image, and output the recognition and detection results;

[0072] S3: The recognition and detection results are fused with the tracking information of the KCF tracker to track a specific target. The tracking state of the KCF tracker is dynamically calibrated according to the recognition and detection results of the YOLO v5n model. At the same time, the recognition and detection frequency of the YOLO v5n model is dynamically adjusted according to the tracking situation of the KCF tracker. The motion state of the target is predicted based on the Kalman filter, and the motion information of the target is output.

[0073] S4: Generate flight control commands based on the target's motion information to drive the UAV to track the target.

[0074] Specifically, in this embodiment, the system hardware and software are first deployed and configured. The drone uses a Raspberry Pi Zero 2 W (quad-core Cortex-A53 CPU, 512MB RAM) as the onboard computing unit, and is equipped with a flight control linkage module, an anomaly detection module, and a PX4 flight controller. The Raspberry Pi Zero 2 W runs Raspberry Pi OS Lite (32-bit simplified system) to reduce resource consumption. Images are acquired through a CSI camera, and the CAP_V4L2 backend of cv2.VideoCapture is used to reduce image acquisition latency. The YOLO v5n model is used as the visual recognition module, and the KCF tracker and Kalman filter are used as the tracking decision module. A "model computational load hierarchical allocation" strategy is adopted: the inference task of the YOLOv5n model is allocated to cores 0-1, the prediction of the KCF tracker and Kalman filter is allocated to core 2, and the flight control communication and command conversion of the flight control linkage module are allocated to core 3. Computational power is balanced through process binding, and the CPU usage is stabilized at 75%-80%. W connects directly to the TELEM2 interface of the PX4 flight controller via the UART pin, and uses the MAVLink protocol for data interaction, skipping redundant forwarding links and improving command transmission speed; the communication baud rate is set to 115200 to ensure the real-time performance and stability of data transmission.

[0075] For specific tracking targets, such as "low, slow, and small" drones, vehicles, and people, a specialized dataset was constructed: Image data of the tracking scene was collected in real time using the CSI camera mounted on the drone. The camera captured high-definition images, which were then transmitted with low latency to the onboard computing unit, a Raspberry Pi Zero 2W, via a CAP_V4L2 backend, ensuring the real-time performance and integrity of the image data. 10,000 images of the tracking target were collected under different environments (sky, buildings, trees, etc.), distances (close range ≤50m, medium range 50-100m, long range >100m), and motion postures (hovering, translation, turning), providing diverse data support for subsequent identification and tracking. The data was labeled using annotation tools, including target bounding boxes with an accuracy of ≤2 pixels and category labels, generating a dataset containing both target bounding boxes and category labels. Data cleaning was performed to remove blurry and severely occluded invalid samples, and the dataset was divided into training and validation sets in an 8:2 ratio to ensure its effectiveness and diversity.

[0076] YOLOv5n (nano version) was selected as the baseline model due to its small parameter and computational cost, making it suitable for the computing power constraints of airborne embedded devices. Mosaic data augmentation was employed, where four different images were randomly scaled by 0.5 to 1.5 times, cropped, horizontally flipped, and then stitched together to increase the diversity of sample backgrounds and the variation in target scale. Copy-Paste data augmentation was also used, where the target region was cropped and randomly pasted onto other backgrounds, with 1 to 2 pasted regions per image, and the IoU between the pasted region and the original target was ≤0.3, thus expanding the number of small target samples. During training, random adjustments were made to... The input image resolution is adjusted to improve the model's adaptability to different distance scales of the target. The K-Means clustering algorithm is used to cluster the ground truth bounding boxes of a specific target dataset, generating nine anchor boxes adapted to specific target sizes: small scale (12×16, 19×36, 40×28), medium scale (36×75, 76×55, 72×146), and large scale (142×110, 192×243, 459×401). The anchor box matching IoU threshold is set to 0.5 to reduce model initialization adaptation costs and accelerate convergence.

[0077] The YOLO v5n model is trained using a combined loss function, the formula for which is as follows:

[0078]

[0079] in, Represents the CIoU loss function. Represents the Focal loss function; Represents the cross-entropy loss function. These correspond to the weights of the three loss functions, respectively. , The weighting ratio is 3:1:1;

[0080] The CIoU loss function, as a position loss function, improves target positioning accuracy by penalizing three dimensions: center point distance, aspect ratio, and overlap area. Its calculation formula is as follows:

[0081] in, This represents a balance factor used to adjust the degree of influence of the aspect ratio term. , Indicates weight, , Indicates the center coordinates of the prediction box ( ), Represents the center coordinates of the true bounding box , This represents the diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted and ground truth bounding boxes. This represents the squared Euclidean distance between the predicted bounding box and the ground truth bounding box.

[0082] The Focal loss function, used as a confidence loss function, addresses the issues of insufficient samples for specific targets and imbalanced positive and negative samples, improving the recall rate for small targets and difficult samples. Its calculation formula is as follows:

[0083]

[0084] in, This represents the positive and negative sample balance coefficient. , This represents the hard sample mining coefficient. , This represents the model's predicted probability for the target class. , This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. These are the true labels of the samples. This represents a positive sample, i.e., the target. This represents a negative sample, i.e., background or interference. This represents the original probability that the target of the model output is the positive class.

[0085] The cross-entropy loss function, used as a category loss function, ensures the accuracy of a specific target category. Its calculation formula is as follows:

[0086] in, It's a real label. This represents the probability distribution predicted by the model, where n is the number of classification categories. An index representing the category.

[0087] After training, the YOLO v5n model achieves an accuracy of FP16. Static quantization is then performed on the trained YOLO v5n model. The FP16 accurate YOLO v5n model undergoes structured layer pruning with a pruning ratio ≤20%, removing redundant channels from each convolutional layer. PyTorch's `torch.quantization` tool is used to insert quantization / dequantization nodes into the pruned model. A portion of the validation set is selected as the calibration dataset for calibration. The model with inserted quantization / dequantization nodes is calibrated, and the numerical distribution information of the input and output of each layer is collected. Based on the calibration results, the layers that have the highest impact on model accuracy are labeled as quantization-sensitive layers. FP16 accuracy is retained for quantization-sensitive layers, while non-sensitive layers are converted to INT8 accuracy, generating a static lightweight model that reduces memory usage and computation while controlling accuracy loss. During deployment, the input resolution is fixed, enabling real-time recognition on airborne equipment.

[0088] Soft-NMS (Soft Non-Maximum Suppression) is used instead of traditional NMS. Instead of directly deleting predicted boxes with an overlap greater than a threshold with the baseline bounding box, its confidence is reduced proportionally to the area of ​​inversion of unity (IoU), reducing false negatives in dense target scenes while retaining valid targets. A confidence threshold (≥0.8) and a target area threshold are set to filter low-confidence predicted boxes and small-area interference objects, reducing the false positive rate. The specific steps are as follows:

[0089] 1) Parameter settings: Preset confidence threshold (≥0.8), IoU threshold (0.45), and Shape threshold (0.3). Let the candidate box set output by the YOLOv5n model be A1, and the corresponding confidence set be S1.

[0090] 2) Initial screening: Select the candidate box B with the highest confidence score F in the confidence score set S1 and add it to the prediction box set A2. At the same time, remove the candidate box B and F from the candidate box set A1 and the confidence score set S1.

[0091] 3) Shape value calculation and confidence adjustment: For the remaining candidate box B in the candidate box set A1, calculate the shape value of B. If the shape value is greater than the shape threshold, reduce the confidence; otherwise, keep the confidence unchanged.

[0092] 4) Iterative filtering: Repeat steps 2 to 3 until only one candidate box remains in A1. Add it to the prediction box set A2, then filter the candidate boxes in the prediction box set A2 whose confidence is lower than the threshold to obtain the second prediction box set A3.

[0093] 5) Redundant box fusion: Calculate the overlap between any two candidate boxes in the second predicted box set A3. If the overlap is higher than the IoU threshold, fuse the bounding box coordinates by area weighting. The confidence of the fused box is taken as the maximum of the two. Finally, output the target box without redundancy, reduce the missed detections and false detections in dense target scenes, and retain the effective targets.

[0094] The YOLO v5n model, after being lightened, is used to perform intermittent target recognition and detection on the image, and the recognition and detection results are output.

[0095] The KCF (Kernelized Correlation Filter) tracker is selected as the core tracking algorithm. Based on cyclic matrix and kernel function optimization, it has low computational cost and high speed, matching the recognition speed of the lightweight YOLO v5n model. A fusion mechanism of "lightweight YOLO v5n model recognition and detection + KCF tracking + Kalman prediction" is constructed to track specific targets. The tracking state of the KCF tracker is dynamically calibrated based on the recognition and detection results of the YOLO v5n model. Simultaneously, the recognition and detection frequency of the YOLO v5n model is dynamically adjusted based on the tracking performance of the KCF tracker. Finally, the motion state of the target is predicted based on the Kalman filter, outputting the target's motion information. The specific collaborative logic is as follows:

[0096] The KCF tracker is initialized with the recognition results of the first full-image detection using the YOLO v5n model;

[0097] In the normal tracking phase, the target tracking position is output frame by frame only through the KCF tracker, while the target position data of the most recent preset number of frames is cached.

[0098] After a preset interval of image frames, the YOLO v5n model is triggered to perform full-image detection again. Based on the YOLO v5n model's detection results, the KCF tracker's tracking is corrected. During correction, the Intersection over Union (IoU) value between the YOLO v5n model's detection bounding box and the KCF tracker's tracking bounding box is calculated. If the IoU value is greater than or equal to a first matching threshold, tracking is considered stable, and the KCF tracker's tracking results are calibrated using the YOLO v5n model's detection results. If the IoU value is less than the first matching threshold but greater than a second matching threshold, slight drift is identified, the KCF tracker's tracking results are retained, and the next frame triggers early execution of the YOLO v5n model's full-image detection. If the IoU value is less than or equal to the second matching threshold, severe drift or target switching is identified, the current KCF tracker's tracking is terminated, and the KCF tracker is reinitialized using the YOLO v5n model's detection results. After correction, the system returns to the normal tracking phase to continue tracking. For example, after system initialization, the YOLO v5n model is executed first in the first frame. The YOLO v5n model performs full-image detection and outputs the initial position of the target with bounding box coordinates, category, and confidence score, which serves as the initialization input for the KCF tracker. Frames 2-5 execute real-time tracking using the KCF tracker while simultaneously caching target position data. In frame 6, the YOLO v5n model performs full-image detection again. If IoU ≥ 0.6, the detection box is used to calibrate the tracking box. If 0.3 < IoU < 0.6, the tracking results of the KCF tracker are retained, triggering early execution of the YOLO v5n model full-image detection in the next frame. If IoU ≤ 0.3, the current KCF tracker's tracking is terminated, and the KCF tracker is reinitialized using the detection results of the YOLO v5n model. Furthermore, the detection interval of the YOLO v5n model is dynamically adjusted based on the target bounding box area (S): if S > 5000 pixels (close range), detection occurs every 7 frames; if 2000 ≤ S ≤ 5000 pixels (medium range), detection occurs every 5 frames; and if S < 2000 pixels (long range), detection occurs every 3 frames, balancing real-time performance and computational cost. If a tracking anomaly is triggered, such as occlusion or low confidence, the fixed timing is broken, and the YOLO v5n model performs full-image detection immediately. After target relocalization, the cyclic timing is restored.

[0099] The visual recognition module, composed of the YOLO v5n model, and the tracking decision module, composed of the KCF tracker, interact via a binary compact encoded data interface. Fields include frame number (4 bytes), target ID (2 bytes), bounding box coordinates (4×2 bytes), confidence level (1 byte, quantized to 0~255), and timestamp (4 bytes). A single frame contains ≤32 bytes of data, and the transmission latency is ≤5ms. Valid recognition results (confidence level ≥0.8) output by the YOLO v5n model are pushed to the KCF tracker in real time through this interface; the real-time tracking results output by the KCF tracker are also synchronized to the flight control linkage module and the anomaly detection module through the same interface format.

[0100] A synchronization mechanism is established: a circular buffer is created as a synchronization cache pool for the YOLO v5n model and the KCF tracker, storing the recognition and detection data of the YOLO v5n model and the tracking data of the KCF tracker for the most recent preset number of image frames; a synchronization flag is set to indicate the data synchronization status, and multi-threaded data reading and writing are performed through atomic operations; the synchronization flag is updated based on the IoU value, and the latest data status in the cache pool is updated synchronously. For example, the recognition and tracking data of the most recent 3 frames are stored, and a synchronization flag is set, where 0 indicates no synchronization, 1 indicates synchronized, and 2 indicates an error. Multi-threaded data reading and writing safety is achieved through atomic operations. When the YOLO v5n model completes the recognition of the Nth frame, it is automatically compared with the tracking result of the Nth frame output by the KCF tracker: if the IoU ≥ 0.6, the synchronization status is determined to be successful, the cache pool status is updated to "synchronized", and the KCF is calibrated with the recognition result of the YOLO v5n model. The tracking bounding box of the tracker; if 0.3 < IoU < 0.6, it is judged as slight drift, the status is marked as "not synchronized", the tracking result of the KCF tracker is retained but the recognition and detection of the YOLO v5n model is triggered in advance in the next frame; if IoU ≤ 0.3, it is judged as severe drift or target switching, the status is marked as "abnormal", the current tracking of the KCF tracker is immediately terminated, and the KCF tracker is reinitialized with the recognition result of the YOLO v5n model. The synchronization determination time is ≤ 1ms.

[0101] Based on the bounding box information of the tracked target output by the KCF tracker, the center coordinates of the tracked target in each frame are extracted. ) and bounding box area (S); calculate the change in target center coordinates over multiple consecutive frames using the sliding window method ( ) and area change rate ( ), combined with time intervals, to obtain the target's two-dimensional velocity ( ) and the rate of scale change ( The formula for the change in center coordinates is: , ,in, Represents the coordinates of the target center in the current frame. The formula for the rate of change of the bounding box area represents the coordinates of the target center in the initial frame of the window. ,in, This represents the area of ​​the target bounding box in the current frame. Indicates the first The width of the frame target bounding box, Indicates the first The height of the target bounding box in the frame. , representing the area of ​​the target bounding box in the initial frame of the window, and the formula for the two-dimensional motion velocity is: ,in The scale change rate formula represents the time interval between adjacent frames. Where N represents the number of consecutively selected frames in the sliding window method. A target motion model is established based on a Kalman filter to predict the position of the tracked target in the next 1-2 frames and output the target's motion information. The state equation of the Kalman filter is: The observation equation is ,in, Indicates the discrete time step or image frame number. Represents the state vector. ,in, This indicates the center coordinates of the target in the image coordinate system. Indicates the target is The speed of movement in the direction, This represents the transpose operation of a vector, where A is the state transition matrix, B is the control input matrix, and B=0. Represents the control input vector. The process noise of the Kalman filter follows a Gaussian distribution. , Indicates a normal distribution. The variance is set as H represents the observation matrix, used to map the state vector to the observation space. The observation noise is represented by a Gaussian distribution. , Indicates a normal distribution. The variance is set as Based on this motion model, the position of the tracked target in the next 1-2 frames is predicted, providing advance for the generation of flight control commands, compensating for recognition delay, with prediction time ≤2ms / frame, and outputting the target's motion information;

[0102] If the tracking bounding box does not update for 3 consecutive frames, the confidence level is lower than the set threshold, or the target motion vector changes abruptly, it is determined that the target is occluded or lost. At this time, the YOLO v5n model is immediately triggered to perform full-image detection and relocate the target. If the target is not detected for several consecutive frames, a "target lost" signal is output.

[0103] Based on the target's motion information, flight control commands are generated to drive the UAV to track the target, and the target center coordinates in the image pixel coordinate system are determined. ),speed( Combining camera parameters and drone altitude information, the target's relative position is converted into the drone's NED (North-East-Down) coordinate system. The calculation formula is:

[0104] , , Where H represents the drone's current altitude above the ground. , This represents the coordinates of the camera's optical center in the pixel coordinate system. , Representing the camera Focal length of direction, , Represents the camera pixel scale factor. Represents the target's altitude above the ground; the converted target's relative position ( The input cascade PID controller's position loop, the outer position loop is relative to the target position ( ) is the input, and the output speed command is ( The inner velocity loop is implemented by the PX4 flight controller's built-in PID controller, which converts speed commands into motor output signals; dynamically adjusts the position loop PID parameters: increasing the proportional coefficient Kp at long distances to accelerate the approach speed; decreasing the proportional coefficient Kp at close distances to avoid oscillations and generate speed commands; filtering the speed commands, a "two-stage filtering" strategy is designed. The first stage uses a moving average filter (window size = 5) to smooth the target position information; the second stage uses a low-pass filter (cutoff frequency = 5Hz) to filter high-frequency noise from the speed commands generated by the cascaded PID controller, avoiding sudden speed changes. Thresholds for the UAV's maximum acceleration and rate of change of speed are set to limit the amplitude of sudden changes in flight control commands and ensure stable flight attitude; the target speed predicted by the Kalman filter ( As a feedforward term, it is added to the speed command output by the PID controller to compensate for the target motion in advance and reduce tracking lag.

[0105] Experimental Example: Using a quadcopter drone equipped with a Raspberry Pi Zero 2W and a PX4 flight controller, the YOLO v5n model recognition and detection algorithm and the Meanshift tracking algorithm were used independently (hereinafter referred to as the background technical solution). A deep fusion mechanism of "YOLO v5n model recognition and detection + KCF tracking + Kalman prediction" was implemented with a lightweight processing, and a control strategy of "two-stage filtering + feedforward compensation based on Kalman prediction" was adopted (hereinafter referred to as the present invention). The results were used to track a "low, slow, and small" drone, and the following data comparison table was obtained:

[0106]

[0107] As can be seen from the table above, the present invention has achieved significant and comprehensive improvements over the background technical solutions in terms of performance indicators such as tracking stability, tracking smoothness, and system real-time performance.

[0108] The continuous tracking drift-free time doubled, the success rate in complex scenarios increased by more than 20 percentage points, and the recapture time was reduced by more than 70%, proving that the deep fusion and state synchronization mechanism can effectively suppress drift and deal with interference, greatly enhancing the robustness of the system.

[0109] The maximum oscillation angle of flight attitude was reduced by 50%, and the oscillation frequency was halved, indicating that the smooth control strategy of "dual-stage filtering + feedforward compensation" transforms unstable visual observations into stable flight commands, thereby improving the stability of the tracking image and flight safety.

[0110] End-to-end latency is reduced by more than 40%, and visual recognition and tracking speeds are improved by approximately 25%-50%. This indicates that the "dynamic detection interval" strategy, through intelligent scheduling of computing power and deployment of lightweight models, enables the system to respond to target motion more quickly.

[0111] The memory footprint of the visual model has been reduced from ≥8 MB to ≥4 MB, thus reducing memory usage.

[0112] This invention employs a deep fusion mechanism of "lightweight YOLO v5n model recognition and detection + KCF tracking + Kalman prediction," combined with periodic calibration and state synchronization strategies, to effectively suppress tracking drift caused by factors such as occlusion and deformation. In scenarios with complex backgrounds, slight occlusion, and targets with variable speed movement, the tracking success rate is significantly improved, the continuous tracking drift-free time is extended, and the efficiency of re-capturing targets after loss is enhanced, thus improving the stability and continuity of dynamic target tracking. Through optimization of the YOLOv5n model, from data, loss function, K-Means adaptive anchor boxes to post-processing, the recognition accuracy of small-scale, low-texture targets is significantly improved. The recognition capability for specific targets such as "low, slow, and small" drones and personnel is significantly enhanced, with a substantial reduction in false positive and false negative rates. It can effectively distinguish targets from interference objects and uses hybrid precision static quantization to adapt to low-computing-power airborne devices, combined with a standardized data interface (single frame data volume ≤ 32). The system communicates directly with UART (with a transmission latency of ≤5ms), ensuring high recognition accuracy for "low, slow, and small" targets while keeping memory and CPU usage within the hardware constraints of low-computing-power airborne devices such as the Raspberry Pi Zero2W. It requires no dedicated GPU, has low power consumption, and is suitable for long-term drone operation. A dynamic detection interval mechanism based on target scale is established: the detection frequency is reduced to save computing power when the target is close (large frame area); the detection frequency is increased to ensure accuracy when the target is far away (small frame area) or unstable, achieving an intelligent balance between computing power consumption and tracking performance. A control strategy of "dual-level filtering + feedforward compensation based on Kalman prediction" effectively filters out visual noise, compensates for system latency, and limits command abrupt changes, resulting in stable drone flight attitude tracking, reduced drone flight attitude oscillations, significantly reduced tracking image jitter, and smoother speed and acceleration changes, thus improving the tracking experience.

[0113] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for tracking unmanned aerial vehicle (UAV) targets based on visual recognition, characterized in that, The method includes: Collect image data of drones tracking targets; The YOLO v5n model, after being lightened, is used to perform intermittent target recognition and detection on the image, and the recognition and detection results are output. The identification and detection results are fused with the tracking information of the KCF tracker to track a specific target. The tracking state of the KCF tracker is dynamically calibrated based on the identification and detection results of the YOLOv5n model. At the same time, the identification and detection frequency of the YOLOv5n model is dynamically adjusted based on the tracking situation of the KCF tracker. The motion state of the target is predicted based on the Kalman filter, and the motion information of the target is output. Flight control commands are generated based on the target's motion information to drive the drone to track the target.

2. The UAV target tracking method based on visual recognition according to claim 1, characterized in that: The process of fusing the identification and detection results with the tracking information of the KCF tracker to track a specific target, and dynamically calibrating the tracking state of the KCF tracker based on the identification and detection results of the YOLO v5n model, includes: The KCF tracker is initialized with the recognition results of the first full-image detection using the YOLO v5n model; In the normal tracking phase, the target tracking position is output frame by frame only through the KCF tracker, while the target position data of the most recent preset number of frames is cached. After a preset interval of image frames, the YOLO v5n model is triggered to perform full-image detection again. The tracking of the KCF tracker is corrected based on the recognition and detection results of the YOLO v5n model. After the correction is completed, the tracking returns to the normal tracking stage to continue tracking.

3. The UAV target tracking method based on visual recognition according to claim 2, characterized in that, The correction of the KCF tracker's tracking based on the recognition and detection results of the YOLO v5n model includes: Calculate the intersection-over-union (IoU) value between the detection bounding boxes of the YOLO v5n model and the tracking bounding boxes of the KCF tracker; If the IoU value is greater than or equal to the first matching threshold, the tracking is determined to be stable, and the tracking results of the KCF tracker are calibrated using the detection results of the YOLO v5n model. If the IoU value is less than the first matching threshold and greater than the second matching threshold, it is determined to be a slight drift. The tracking result of the KCF tracker is retained, and the YOLO v5n model is triggered to perform full-image detection in the next frame in advance. If the IoU value is less than or equal to the second matching threshold, it is determined to be a severe drift or target switching, the current KCF tracker is terminated, and the KCF tracker is reinitialized using the detection results of the YOLO v5n model.

4. The UAV target tracking method based on visual recognition according to claim 3, characterized in that, When dynamically calibrating the tracking state of the KCF tracker based on the recognition and detection results of the YOLO v5n model: A circular buffer is established as a synchronization cache pool for the YOLO v5n model and the KCF tracker, storing the recognition and detection data of the YOLO v5n model and the tracking data of the KCF tracker for the most recent preset number of image frames; Set a synchronization flag to indicate the data synchronization status, and perform multi-threaded data reading and writing through atomic operations; The synchronization flag is updated based on the IoU value, and the latest data status in the cache pool is updated synchronously.

5. The UAV target tracking method based on visual recognition according to claim 1, characterized in that: The dynamic adjustment of the YOLO v5n model's recognition and detection frequency based on the tracking performance of the KCF tracker includes: Calculate the bounding box area of ​​the target currently being tracked by the KCF tracker; If the bounding box area is greater than the first pixel threshold, then a first detection interval is set; If the bounding box area is less than or equal to the first pixel threshold and greater than the second pixel threshold, then a second detection interval less than the first detection interval is set. If the bounding box area is less than or equal to the second pixel threshold, then a third detection interval is set that is less than the second detection interval.

6. The UAV target tracking method based on visual recognition according to claim 1, characterized in that: The lightweight YOLO v5n model is obtained through the following steps: Collect multi-scene image datasets containing specific targets; The K-Means clustering algorithm is used to perform cluster analysis on the true bounding boxes of targets in the image dataset, generating adaptive anchor box samples that match the target scale distribution. The YOLO v5n model is trained using a combined loss function, the formula for which is as follows: ； in, Represents the CIoU loss function. Represents the Focal loss function; Represents the cross-entropy loss function. These correspond to the weights of the three loss functions, respectively. , The weighting ratio is 3:1:1; Static quantization is performed on the trained YOLO v5n model to convert the non-quantization-sensitive layers in the YOLO v5n model to INT8 precision, while retaining FP16 precision for the quantization-sensitive layers, thus generating a lightweight YOLO v5n model. Soft nonmaximum suppression (Soft-NMS) is used to post-process the predicted bounding boxes output by the lightweight YOLO v5n model inference, reducing missed detections in dense target scenes and retaining valid targets.

7. The UAV target tracking method based on vision recognition according to claim 6, characterized in that: The formula for calculating the CIoU loss function is as follows: ; in, This represents a balance factor used to adjust the degree of influence of the aspect ratio term. Indicates weight, Indicates the center coordinates of the prediction box ( ), Represents the center coordinates of the true bounding box , This represents the diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted and ground truth bounding boxes. This represents the squared Euclidean distance between the predicted bounding box and the ground truth bounding box; The formula for calculating the Focal loss function is as follows: ; in, This represents the positive and negative sample balance coefficient. This represents the hard sample mining coefficient. This represents the model's predicted probability for the target class. , Indicates an indicator function, These are the true labels of the samples. This represents the original probability that the target value output by the model is the positive class. The formula for calculating the cross-entropy loss function is as follows: ; in, These are the true labels of the samples. It is the probability distribution predicted by the model. It is the number of categories. An index representing the category.

8. The UAV target tracking method based on vision recognition according to claim 6, characterized in that: The step of statically quantizing the trained YOLO v5n model, converting the non-quantization-sensitive layers to INT8 precision, and retaining FP16 precision for the quantization-sensitive layers to generate a lightweight YOLO v5n model includes: Structural layer pruning was performed on the trained YOLO v5n model with FP16 accuracy to remove redundant channels in each convolutional layer. Use PyTorch's torch.quantization tool to insert quantization / dequantization nodes into the pruned model; The model with inserted quantization / dequantization nodes is calibrated using a calibration dataset, and the numerical distribution information of the input and output of each layer is collected. Based on the calibration results, the layers that have the greatest impact on the model accuracy are labeled as quantization-sensitive layers. The FP16 accuracy of the quantization-sensitive layers is retained, while the non-sensitive layers are converted to INT8 accuracy, thus generating a static lightweight model.

9. The UAV target tracking method based on vision recognition according to claim 1, characterized in that: The motion state prediction of the target based on the Kalman filter outputs the target's motion information, including: Based on the bounding box information of the tracked target output by the KCF tracker, the center coordinates and bounding box area of ​​the tracked target are extracted; The change in the center coordinates and the rate of change in the bounding box area of ​​the tracked target are calculated using the sliding window method. A target motion model is established based on a Kalman filter to predict the target's position in the next 1-2 frames and output the target's motion information. The state equation of the Kalman filter is: The observation equation is ,in, Indicates the discrete time step or image frame number. Represents the state vector. ,in, This indicates the center coordinates of the target in the image coordinate system. Indicates the target is The speed of movement in the direction, This represents the transpose operation of a vector, where A is the state transition matrix, B is the control input matrix, and B=0. Represents the control input vector. The process noise of the Kalman filter follows a Gaussian distribution. , Indicates a normal distribution. The variance is set as H represents the observation matrix, used to map the state vector to the observation space. The observation noise is represented by a Gaussian distribution. , Indicates a normal distribution. The variance is set as .

10. The UAV target tracking method based on visual recognition according to claim 1, characterized in that: The step of generating flight control commands based on the target's motion information to drive the UAV to track the target includes: The target position and velocity information in the image coordinate system are combined with the drone altitude and camera parameters to transform the target relative position in the drone body coordinate system. The deviation of the target relative position after conversion is input into the position loop of the cascade PID controller to generate a speed command; The speed command is filtered, and the target speed predicted by the Kalman filter is used as a feedforward quantity for superposition compensation.

11. The UAV target tracking method based on visual recognition according to claim 10, characterized in that... The speed command is filtered, and the target speed predicted by the Kalman filter is used as a feedforward for superposition compensation, including: The target position information is smoothed by applying a moving average filter. The speed command generated by the cascaded PID controller is low-pass filtered to filter out high-frequency noise. Set the maximum acceleration and rate of change thresholds for the drone to limit the abrupt changes in flight control commands; The target velocity predicted by the Kalman filter is used as a feedforward quantity and added to the velocity command to compensate for the target motion in advance.

Images