Unmanned aerial vehicle cruise target detection and tracking method

Abstract

The application discloses a cruise target detection and tracking method based on a UAV, and comprises the following steps: acquiring an image, performing target detection, and acquiring a target; tracking and positioning the target; establishing a database model for the target; and tracking the target.The cruise target detection and tracking method based on the UAV can continue to be locked and tracked after the tracking target is blocked or after the tracking target is out of the field of view and then returns to the field of view, and the loss of the tracking target is avoided.

Description

Technical Field

[0001] This invention relates to the field of power grid drone patrol technology, and in particular to a method for drone-based target detection and tracking. Background Technology

[0002] Currently, in the power grid sector, there are numerous construction sites, each requiring safety monitoring. This includes monitoring whether construction workers are wearing safety helmets and proper clothing, whether they are smoking in hazardous areas, and preventing accidents such as unauthorized personnel entering energized areas. Traditional monitoring personnel must monitor a large number of camera feeds, making it impossible to simultaneously monitor all work sites in real-time. There is a conflict between the high-intensity monitoring demands and the physical condition of the monitoring personnel. Human monitoring has inherent limitations; human energy is limited. Video surveillance requires monitoring numerous work points and long monitoring times, resulting in a heavy workload. Monitoring personnel work under intense pressure, constantly staring at screens, which inevitably leads to fatigue over time, reducing monitoring efficiency and impacting effectiveness. Furthermore, because traditional cameras are fixed in position, blind spots exist within the monitoring range, preventing comprehensive monitoring of the construction site.

[0003] Using drones for inspections has become an unstoppable trend. However, drone inspections require drones to continuously and automatically detect and track targets during the filming process and issue timely alarms, which is a major challenge in this field.

[0004] Object detection is the task of identifying objects of interest in images or videos, while simultaneously determining their location and size. Unlike image classification, object detection requires solving both classification and localization problems. The development of object detection can be divided into traditional object detection algorithms and deep learning-based object detection algorithms. Traditional object detection approaches include region selection, manual feature extraction, and classifier classification. However, because manual feature extraction methods often struggle to capture the diverse characteristics of objects, traditional methods have consistently failed to adequately solve the object detection problem.

[0005] In recent years, two-stage object detection algorithms have been widely used due to their high detection accuracy. While these methods offer high accuracy, their detection speed is relatively slow, often failing to achieve industrial-grade detection results. In contrast, single-stage object detection algorithms directly calculate and generate detection results from the image, offering faster detection speed but lower accuracy. Despite its lower prediction accuracy compared to two-stage algorithms, YOLO has become the mainstream algorithm in industry due to its faster processing speed.

[0006] Furthermore, after finding the target based on object detection, target tracking is also crucial. Its goal is to track one or more targets in real-time within a video sequence, that is, to locate and track the target's position between consecutive frames. Multi-object tracking (MOT) aims to detect targets of interest or to be tracked within video frames and obtain their positions in the image, assigning each target an ID and maintaining each target ID unchanged during target movement. However, ByteTrack, a mainstream algorithm based on MOT, is a tracking-by-detection paradigm. Most multi-object tracking methods obtain target IDs by associating detection boxes with scores above a threshold. Targets with low detection scores, such as occluded targets, are simply discarded, leading to significant problems, including numerous missed detections and fragmented trajectories.

[0007] Therefore, in the current target tracking process, when the target is obscured or disappears from the field of view for a certain period of time and then reappears, the tracked target will be lost. This problem urgently needs to be solved. Summary of the Invention

[0008] In order to overcome the shortcomings of the prior art, the present invention provides a method for target detection and tracking based on unmanned aerial vehicle (UAV) cruise.

[0009] This invention is achieved through the following technical solution:

[0010] The method for detecting and tracking targets during drone patrols includes the following steps:

[0011] S1: Acquire the image, perform object detection, and acquire the target;

[0012] S2: Track and locate the target;

[0013] S3: Establish a database model for the target;

[0014] S3.1: Obtain the area S occupied by the target in the image, obtain the change curve of the area S based on the change of video frames over time, and determine the distance relationship between the target and the photographer based on the change curve;

[0015] S3.2: Obtain the position of the target in the image, obtain the vector of the position movement based on the change of video frames over time, and determine the direction of movement of the target based on the vector;

[0016] S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of pixel grayscale value of each level;

[0017] S3.4: Obtain the percentage data of different gray values ​​in the area of ​​the target region in the image;

[0018] S3.5: Based on the proportion of grayscale values ​​of each pixel level and the area of ​​the target in the image, and according to the changes in the proportion of different grayscale values ​​of the video frames over time, analyze the motion trajectory of the target and the change curve of the proportion of grayscale values ​​of each pixel level.

[0019] S4: Tracking Target:

[0020] When the target is obscured by other objects and re-enters the field of view, the target is tracked based on its movement trajectory and the change curve of the proportion of different gray values.

[0021] The change curve of the area is used to determine whether the target has moved away from the shooting field of view;

[0022] The target is determined by the vector to determine whether it has left the field of view. When it re-enters the field of view, the target is tracked based on the reverse direction of the vector, the target's motion trajectory, and the change curve of the proportion of different gray values.

[0023] Furthermore, in S1, the target detection includes performing target detection using the YOLO algorithm.

[0024] Furthermore, in S2, the tracking and positioning of the acquired target includes using the ByteTrack algorithm in the MOT algorithm, which is a type of multi-target tracking algorithm.

[0025] Furthermore, the tracking and localization of the acquired target also includes the BoT-SORT multi-target tracking algorithm, which provides camera motion compensation and improves tracking performance.

[0026] Furthermore, step S3.3 also includes: obtaining curves showing the changes in the proportion of pixel grayscale values ​​at each level caused by different backgrounds based on the changes in video frames over time, determining the change curve of the range of pixel grayscale value proportion levels where the target is located, and determining whether the tracked target is the original tracked target based on the change curves.

[0027] Furthermore, step S3.4 also includes: based on the obtained data of the proportion of different gray values ​​in the area of ​​the target in the image, the target with the smallest change in gray value proportion is identified as the target, which is used to determine whether the tracked target is correct.

[0028] Furthermore, step S3.5 also includes: based on the change curve of the pixel grayscale value proportion data of each level, when the change curve of the pixel grayscale value proportion data of the largest proportion level jumps, determining that the target is obscured or removed from the shooting field of view.

[0029] Compared with existing technologies, the beneficial effects of this invention are:

[0030] The UAV-based target detection and tracking method provided in this application can continue to lock onto and track a target even after it is obscured or after it leaves the field of view and then returns to the field of view, without causing the target to be lost. The specific beneficial effects are described in detail in the specific implementation method. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] The method for detecting and tracking targets during drone patrols includes the following steps:

[0033] S1: Acquire image, perform target detection, acquire target, and perform target detection using the YOLO algorithm.

[0034] In recent years, two-stage object detection algorithms have been widely used due to their high detection accuracy. While these methods offer high accuracy, their detection speed is relatively slow, often failing to achieve industrial-grade detection results. In contrast, single-stage object detection algorithms directly calculate and generate detection results from the image, offering faster detection speed but lower accuracy. Despite its lower prediction accuracy compared to two-stage algorithms, YOLO has become the mainstream algorithm in industry due to its faster processing speed.

[0035] The YOLO algorithm's network structure is inspired by GoogLeNet, containing 24 convolutional layers and 2 fully connected layers. First, an image is input and divided into an n×n grid. For each grid, b bounding boxes are predicted. Then, n×n×b target windows are predicted. Next, low-probability target windows are removed based on a set threshold. Finally, redundant windows are removed using Non-Maximum Suppression (NMS) to obtain the detection result.

[0036] S2: Track and locate the target. Tracking and locating the acquired target includes using the ByteTrack algorithm in the MOT algorithm. The ByteTrack algorithm is a type of multi-target tracking algorithm.

[0037] The target tracking and localization process also includes the BoT-SORT multi-target tracking algorithm, which provides camera motion compensation and improves tracking performance.

[0038] Multi-object tracking (MOT) aims to detect and locate targets of interest or to be tracked within video frames, assigning each target an ID that remains constant throughout the target's motion. However, ByteTrack, a mainstream algorithm within MOT, is based on a tracking-by-detection paradigm. Most MOT methods obtain target IDs by associating detection boxes with scores above a threshold. Targets with low detection scores, such as occluded targets, are simply discarded, leading to significant problems including numerous missed detections and fragmented trajectories. To address this issue, the authors propose a simple, efficient, and general data association method, BYTE, which tracks targets by associating each detection box, not just high-scoring ones. For low-scoring detection boxes, their similarity to the trajectory is used to reconstruct the true target and filter out background detections.

[0039] Because ByteTrack employs a detector-based tracking framework, it requires re-detecting the target in each frame and then tracking it based on the detection results. Such detectors are typically complex, potentially posing computational challenges for real-time applications or low-power devices. We address this by adopting the BoT-SORT multi-target tracking algorithm with motion compensation. Building upon ByteTrack, we add camera motion compensation, thereby improving the performance of the detector-based tracker.

[0040] Unlike the ByteTrack tracking algorithm, the BoT-SORT algorithm primarily adds a Kalman filter region to fit the detected target width. It also introduces camera motion compensation (CMC) to optimize the jitter problem during dynamic display. In SORT, the state variable is... 's' is the box's dimensions (or area), and 'a' is the box's aspect ratio. Experiments have shown that directly outputting the dimensions yields better results. Therefore, using...

[0041]

[0042] Represents state variables,

[0043] z k =[z xc (k),z yc (k),z w (k),z h (k)]

[0044] This represents the observed value.

[0045] In SORT, Q and R are time-independent, but in DeepSort, they are correlated with both measured and observed values. Therefore, the improved Kalman filter results in higher accuracy. Tracking-by-detection trackers heavily rely on the overlap between the predicted and detected bounding boxes. In dynamic camera scenarios, the pixel positions of the boxes can change drastically, leading to tracking loss. Even in static cameras, slight vibrations due to wind can cause problems. Therefore, a camera motion compensation model is introduced to address this vibration issue. Without camera parameters, registration between adjacent frames can approximate camera motion. OpenCV's global motion compensation (GMC) can be used. First, feature points are extracted from the image. Then, outliers are filtered out based on sparse optical flow, and finally, RANSAC is used to calculate the radiometric changes. Sparse matching can eliminate moving parts (obtained through detection), thus achieving more accurate background matching.

[0046] During use Transform the trajectory prediction box of frame k-1 to frame k. The translation in the transformation matrix applies to the center point. As shown in the following equation:

[0047]

[0048] S3: Establish a database model for the target.

[0049] S3.1: Obtain the area occupied by the target in the image, obtain the change curve of the area based on the video frame over time, and determine the distance relationship between the target and the photographer based on the change curve.

[0050] As the video frames progress through time, the target moves within the image. If it moves away from the photographer, its area decreases; if it moves in a certain direction, its area increases. Similarly, if the target remains stationary and the photographer moves closer, its area increases; if the target remains stationary and the photographer moves away, its area decreases. By observing the changes in area over time, a time curve of area change can be obtained. The changes in the curve can then be used to determine the relationship between the distance between the target and the photographer.

[0051] S3.2: Obtain the position of the target in the image, obtain the vector of the position movement based on the change of video frames over time, and determine the direction of movement of the target based on the vector.

[0052] S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of pixel grayscale value of each level.

[0053] For example, the grayscale values ​​are divided into 10 levels, and the percentage of pixel grayscale values ​​in each level is calculated.

[0054] Based on the changes in video frames over time, obtain the curves of changes in the proportion of pixel grayscale values ​​at each level caused by different backgrounds, determine the curves of changes in the range of pixel grayscale value proportion levels where the target is located, and determine whether the tracked target is the original tracked target based on the curves of changes.

[0055] For example, the proportions of the 10 levels of pixel grayscale values ​​form 10 variation curves. Based on the smoothness of the curve changes, it is determined that some levels of pixel grayscale values ​​include pixel grayscale values ​​belonging to the tracking target. The specific determination method is described in detail below.

[0056] S3.4: Obtain the percentage data of different gray values ​​in the area of ​​the target region in the image.

[0057] Based on the obtained data of the proportion of different gray values ​​in the area of ​​the target in the image, the target with the smallest change in gray value proportion is identified as the target, which is used to determine whether the tracked target is correct.

[0058] S3.5: Based on the proportion of grayscale values ​​of each pixel level and the area of ​​the target in the image, and according to the changes in the proportion of different grayscale values ​​of the video frames over time, analyze the motion trajectory of the target and the change curve of the proportion of grayscale values ​​of each pixel level.

[0059] Based on the change curve of the pixel grayscale value proportion data of each level, when the change curve of the pixel grayscale value proportion data of the largest proportion level jumps, it is determined that the target is obscured or removed from the shooting field of view.

[0060] S4: Tracking Target:

[0061] When the target is obscured by other objects and re-enters the field of view, the target is tracked based on its movement trajectory and the change curve of different grayscale values.

[0062] The change curve of the area is used to determine whether the target has moved away and disappeared from the field of view.

[0063] The target is determined by the vector Z to determine whether it has left the field of view. When it re-enters the field of view, the target is tracked based on the reverse direction of the vector Z, the target's motion trajectory, and the change curve of the proportion of different gray values.

[0064] Compared with existing technologies, the beneficial effects of this invention are:

[0065] The UAV-based target detection and tracking method provided in this application can continue to lock onto and track a target even after it is obscured or after it leaves the field of view and then returns to the field of view, without causing the target to be lost. The specific beneficial effects are described in detail in the specific implementation method.

[0066] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for target detection and tracking based on unmanned aerial vehicle (UAV) cruise, characterized in that, Includes the following steps: S1: Acquire the image, perform object detection, and acquire the target; S2: Track and locate the target; S3: Establish a database model for the target; S3.1: Obtain the area occupied by the target in the image, obtain the change curve of the area based on the change of video frames over time, and determine the distance relationship between the target and the photographer based on the change curve; S3.2: Obtain the position of the target in the image, obtain the vector of the position movement based on the change of video frames over time, and determine the direction of movement of the target based on the vector; S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of pixel grayscale value of each level; S3.4: Obtain the percentage data of different gray values ​​in the area of ​​the target region in the image; S3.5: Based on the proportion of grayscale values ​​of each pixel level and the area of ​​the target in the image, and according to the changes in the proportion of different grayscale values ​​of the video frames over time, analyze the motion trajectory of the target and the change curve of the proportion of grayscale values ​​of each pixel level. S4: Tracking Target: When the target is obscured by other objects and re-enters the field of view, the target is tracked based on its movement trajectory and the change curve of the proportion of different gray values. The change curve of the area is used to determine whether the target has moved away from the shooting field of view; The target is determined by the vector to determine whether it has left the field of view. When it re-enters the field of view, the target is tracked based on the reverse direction of the vector, the target's motion trajectory, and the change curve of the proportion of different gray values.

2. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 1, characterized in that, In S1, the target detection includes performing target detection using the YOLO algorithm.

3. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 1 or 2, characterized in that, In step S2, the acquired target is tracked and located, including using the ByteTrack algorithm in the MOT algorithm, which is a type of multi-target tracking algorithm.

4. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 3, characterized in that, The target tracking and localization process also includes the BoT-SORT multi-target tracking algorithm, which provides camera motion compensation and improves tracking performance.

5. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 1, characterized in that, The S3.3 step further includes: obtaining the curve of the change in the proportion of pixel grayscale value of each level caused by different backgrounds based on the change of video frames over time, determining the change curve of the range of pixel grayscale value proportion level of the target, and determining whether the tracked target is the original tracked target based on the change curve.

6. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 1, characterized in that, Step S3.4 further includes: based on the obtained data of the proportion of different gray values ​​in the area of ​​the target in the image, the target with the smallest change in gray value proportion is identified as the target, which is used to determine whether the tracked target is correct.

7. The method for detecting and tracking unmanned aerial vehicle (UAV) cruise targets according to claim 1, characterized in that, The S3.5 step further includes: based on the change curve of the pixel grayscale value proportion data of each level, when the change curve of the pixel grayscale value proportion data of the largest proportion level jumps, determining that the target is obscured or removed from the shooting field of view.

Images