Target detection and tracking method based on unmanned aerial vehicle cruise

Abstract

The invention discloses an unmanned aerial vehicle-based cruise target detection and tracking method, which comprises the following steps of: acquiring an image, performing target detection and acquiring a target; tracking and positioning the target; establishing a database model for the target; and tracking the target. According to the unmanned aerial vehicle-based cruising target detection and tracking method provided by the invention, the tracking target can be locked and tracked continuously after being shielded or being separated from the view field and returning to the view field again, and the tracking target cannot be lost.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid unmanned aerial vehicle cruise, and in particular to a target detection and tracking method based on unmanned aerial vehicle cruise. Background Art

[0002] At present, in the field of power grid, there are many construction sites, and each site needs safety monitoring. For example, whether the construction workers wear safety helmets, whether the clothes are standardized, whether they smoke in dangerous places, and other violations, as well as accidents such as non-staff entering the live area of ​​the construction site. Traditional monitoring personnel have to monitor a large number of camera videos, and cannot achieve real-time monitoring of all the work sites at the same time. The contradiction between the high-intensity monitoring demand and the status of the monitoring personnel themselves. Personnel monitoring itself has shortcomings. People have limited energy. Video monitoring requires monitoring of many work points, long monitoring time, and heavy monitoring tasks. Monitoring personnel work in a state of staring at the screen and concentrating their energy. After a long time, fatigue is inevitable, and the monitoring efficiency decreases, affecting the monitoring effect. At the same time, due to the fixed position of traditional cameras, there are visual blind spots in the monitoring range, and the construction site cannot be fully monitored.

[0003] Inspections using drones have become an unstoppable trend. However, inspections using drones require drones to continuously and automatically detect and track targets during the shooting process and issue timely alarms, which is a major challenge in this field.

[0004] The task of object detection is to find objects of interest in images or videos, and detect their positions and sizes at the same time. Unlike image classification tasks, object detection not only solves classification problems, but also positioning problems. The development of object detection can be divided into traditional object detection algorithms and object detection algorithms based on deep learning. Traditional object detection ideas include region selection, manual feature extraction, and classifier classification. Since manual feature extraction methods often find it difficult to meet the diverse characteristics of the target, traditional methods have never been able to solve the object detection problem well.

[0005] In recent years, the two-stage target detection algorithm has been widely used due to its high detection accuracy. Although this type of method has high detection accuracy, it has a slow detection speed and cannot achieve industrial detection results in many cases. However, compared with the two-stage target detection algorithm, the single-stage target detection algorithm directly calculates the image to generate detection results. The detection speed is fast, but the detection accuracy is low. Although the prediction accuracy is not as good as the two-stage target detection algorithm, due to its faster running speed, YOLO has become the mainstream in the industry.

[0006] In addition, after finding the target based on target detection, target tracking is also crucial. Its goal is to track one or more targets in real time in a video sequence, that is, to locate and track the position of the target between consecutive frames. Multi-target tracking technology (MOT) aims to detect the targets of interest or desired tracking in the video frame and obtain their positions in the image, and assign an ID to each target. During the movement of the target, each target ID remains unchanged. However, ByteTrack in the MOT algorithm is the mainstream algorithm based on the multi-target algorithm. ByteTrack is a tracking method based on the tracking-by-detection paradigm. Most multi-target tracking methods obtain the target ID by associating detection boxes with scores higher than a threshold. For targets with low detection scores, such as occluded targets, they are simply discarded, which brings non-negligible problems, including a large number of missed detections and fragmented trajectories.

[0007] Therefore, in the current target tracking process, when the target is blocked or leaves the field of view for a certain period of time and then reappears, the tracking target will be lost. This problem needs to be solved urgently. Summary of the invention

[0008] In order to overcome the deficiencies in the prior art, the present invention provides a method for detecting and tracking a cruise target based on an unmanned aerial vehicle.

[0009] The present invention is achieved through the following technical solutions:

[0010] The method for detecting and tracking a target based on a UAV cruise includes the following steps:

[0011] S1: Acquire images, perform target detection, and acquire targets;

[0012] S2: Tracking and positioning the target;

[0013] S3: Establishing a database model for the target;

[0014] S3.1: Obtain the area S occupied by the target in the image, obtain a change curve of the area S according to the change of the video frame over time, and determine the distance relationship between the target and the photographer according to the change curve;

[0015] S3.2: Obtain the position of the target in the image, obtain a vector of movement of the position according to changes in the video frame over time, and determine the moving direction of the target according to the vector;

[0016] S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of the pixel grayscale value of each level;

[0017] S3.4: Obtaining data on the proportion of different grayscale values ​​in the area of ​​the target in the image;

[0018] S3.5: According to the proportion of the grayscale value of each level of pixels and the area of ​​the target in the image, according to the change of the video frame over time, the proportion of different grayscale values ​​​​area analyzes the movement trajectory of the target and the change curve of the proportion of the grayscale value of each level of pixels;

[0019] S4: Tracking target:

[0020] When the target is blocked by other objects and re-enters the shooting field of view, the target is tracked according to the moving trajectory of the target and the change curve of the proportion of different gray values;

[0021] Judging whether the target has moved away and is out of the shooting field of view by the change curve of the area of ​​the region;

[0022] The vector is used to determine whether the target is out of the shooting field of view. When the target re-enters the field of view, the target is tracked according to the reverse direction of the vector, the target's motion trajectory, and the change curve of the proportion of different grayscale values.

[0023] Furthermore, in S1, the target detection includes using a YOLO algorithm to perform target detection.

[0024] Furthermore, in S2, tracking and locating the acquired target includes adopting the ByteTrack algorithm in the MOT algorithm, and the ByteTrack algorithm is a multi-target tracking algorithm.

[0025] Furthermore, the tracking and positioning of the acquired target also includes a BoT-SORT multi-target tracking algorithm to provide camera motion compensation and improve tracking performance.

[0026] Furthermore, the S3.3 step also includes: obtaining a curve of the change in the proportion of pixel grayscale values ​​at each level caused by different backgrounds based on the change of the video frame over time, determining a curve of the change in the range of the pixel grayscale value proportion level of the target, and determining whether the tracked target is the original tracked target based on the change curve.

[0027] Furthermore, the step S3.4 also includes: based on the data of the proportion of different grayscale values ​​in the area of ​​the target in the image, the target with the smallest change in grayscale value proportion is identified as the target, which is used to determine whether the tracked target is correct.

[0028] Furthermore, the S3.5 step also includes: according to the change curve of the pixel grayscale value proportion data of each level, when the change curve of the pixel grayscale value proportion data of the maximum proportion level jumps, it is determined that the target is blocked or out of the shooting field of view.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The drone cruise target detection and tracking method provided in the present application can continue to be locked and tracked after the tracking target is blocked, or after it leaves the field of view and then returns to the field of view, without causing the loss of the tracking target. The specific beneficial effects are described in detail in the specific implementation method. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The method for detecting and tracking a target based on a UAV cruise includes the following steps:

[0033] S1: Acquire an image, perform target detection, acquire a target, and perform target detection using the YOLO algorithm.

[0034] In recent years, the two-stage target detection algorithm has been widely used due to its high detection accuracy. Although this type of method has high detection accuracy, it has a slow detection speed and cannot achieve industrial detection results in many cases. However, compared with the two-stage target detection algorithm, the single-stage target detection algorithm directly calculates the image to generate detection results. The detection speed is fast, but the detection accuracy is low. Although the prediction accuracy is not as good as the two-stage target detection algorithm, due to its faster running speed, YOLO has become the mainstream in the industry.

[0035] The network structure of the YOLO algorithm is inspired by GoogLeNet, which contains 24 convolutional layers and 2 fully connected layers. First, an image is input and divided into an n×n grid map. For each grid, b bounding boxes are predicted. Then, n×n×b target windows can be predicted, and then target windows with low probability are removed according to the set threshold. Finally, redundant windows are removed through NMS to obtain the detection result.

[0036] S2: Tracking and locating the target. Tracking and locating the acquired target includes adopting the ByteTrack algorithm in the MOT algorithm. The ByteTrack algorithm is a multi-target tracking algorithm.

[0037] The tracking and positioning of the acquired target also includes a BoT-SORT multi-target tracking algorithm to provide camera motion compensation and improve tracking performance.

[0038] Multi-object tracking (MOT) aims to detect the objects of interest or desired tracking in the video frame and obtain their positions in the image, and assign an ID to each object. During the movement of the object, the ID of each object remains unchanged. However, ByteTrack in the MOT algorithm is a mainstream algorithm based on multi-object algorithms. ByteTrack is a tracking method based on the tracking-by-detection paradigm. Most multi-object tracking methods obtain the target ID by associating detection boxes with scores higher than a threshold. For objects with low detection scores, such as occluded objects, they are simply discarded, which brings non-negligible problems, including a large number of missed detections and fragmented trajectories. To solve this problem, the author proposed a simple, efficient and general data association method BYTE, which tracks by associating each detection box instead of just the high-scoring detection boxes. For low-scoring detection boxes, their similarity with the trajectory is used to recover the true target and filter out background detections.

[0039] Since ByteTrack uses a detector-based tracking framework, it needs to re-detect the target in each frame and then track the target based on the detection results. Such detectors are usually complex and may have computational complexity issues for real-time applications or low-power devices. We adopt the BoT-SORT multi-target tracking algorithm with motion compensation, and add a camera motion compensation on the basis of ByteTrack, which improves the performance of the detector-based tracker.

[0040] Different from the ByteTrack tracking algorithm, the BoT-SORT algorithm mainly adds a Kalman filter area to fit the detected target width. It also introduces camera motion compensation (CMC) to optimize the jitter problem during dynamic display. In SORT, the state variable is s is the size (or area) of the box, and a is the aspect ratio of the box. Through experiments, it is found that directly outputting the length and width will have a better effect.

[0041]

[0042] represents the state variable,

[0043] z k =[z xc (k),z yc (k),z w (k),z h (k)]

[0044] Represents the observed value.

[0045] In SORT, Q and R are time-independent, but in DeepSort they are changed to be related to the measured and observed values. Therefore, the improved Kalman filter brings higher accuracy. Since the Tracking-by-detection tracker relies heavily on the overlap between the trajectory prediction box and the detection box. In a dynamic camera scene, the pixel position of the box will change dramatically, resulting in loss of tracking. Even in a static camera, slight vibrations can be caused by the influence of wind. Therefore, the camera motion compensation model method is introduced to solve the vibration problem. In the absence of camera parameters, the registration between adjacent frames can approximate the movement of the camera. You can use OpenCV's global motion compensation (GMC). First, feature points are extracted from the image, then outliers are filtered out based on the sparse optical flow, and then RANSAC is used to calculate the radiation changes. Sparse matching can remove moving parts (obtained through detection) to obtain more accurate background matches.

[0046] During use Transform the trajectory prediction frame of frame k-1 to frame k. The translation in the transformation matrix acts on the center point. As shown in the following formula:

[0047]

[0048] S3: Establish a database model for the target.

[0049] S3.1: Obtain the area of ​​the region occupied by the target in the image, obtain a change curve of the area of ​​the region according to the change of the video frame over time, and determine the distance relationship between the target and the photographer according to the change curve.

[0050] The target in the image will move as the video frame continues to advance along time. If it moves away from the photographer, the area of ​​the target will become smaller. If the target moves in the direction, the area of ​​the target will become larger. Similarly, if the target is stationary and the photographer approaches the target, the area of ​​the target will become larger. If the target is stationary and the photographer moves away from the target, the area of ​​the target will become smaller. Therefore, according to the change of the area over time, a time curve graph of the change of the area can be obtained, and the changing relationship of the distance between the target and the photographer can be judged according to the change of the curve.

[0051] S3.2: Obtain the position of the target in the image, obtain the vector of the movement of the position according to the change of the video frame over time, and determine the moving direction of the target according to the vector.

[0052] S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of the pixel grayscale value of each level.

[0053] Exemplarily, the grayscale value is divided into 10 levels, and the proportion of the grayscale value of pixels in each level is counted.

[0054] According to the changes of the video frame over time, a curve of the change of the grayscale value proportion of each level of pixels caused by different backgrounds is obtained, and a change curve of the range of the grayscale value proportion level of the target is determined, and whether the tracked target is the original tracked target is determined according to the change curve.

[0055] Exemplarily, the proportions of the 10 levels of pixel grayscale values ​​form 10 change curves. According to the smoothness of the curve changes, it is determined that several levels of pixel grayscale values ​​include the pixel grayscale belonging to the tracking target. The specific determination method is described in detail below.

[0056] S3.4: Obtaining data on the proportion of different grayscale values ​​in the area of ​​the target in the image.

[0057] According to the data of proportions of different gray values ​​in the area of ​​the target in the image, the target with the smallest change in gray value proportion is identified as the target, which is used to determine whether the tracked target is correct.

[0058] S3.5: According to the proportion of the grayscale value of each level of pixels and the area of ​​the target in the image, according to the change of the video frame over time, the proportion data of different grayscale values ​​are analyzed to analyze the movement trajectory of the target and the change curve of the proportion data of the grayscale value of each level of pixels.

[0059] According to the variation curve of the pixel gray value proportion data of each level, when the variation curve of the pixel gray value proportion data of the maximum proportion level jumps, it is determined that the target is blocked or out of the shooting field of view.

[0060] S4: Tracking target:

[0061] When the target is blocked by other objects and re-enters the shooting field of view, the target is tracked according to the movement trajectory of the target and the change curve of the proportion data of different gray values.

[0062] It is determined whether the target has moved away and is out of the shooting field of view through the change curve of the area of ​​the region.

[0063] The vector Z is used to determine whether the target is out of the shooting field of view. When the target re-enters the field of view, the target is tracked according to the reverse direction of the vector Z, the target's motion trajectory, and the change curve of the proportion of different grayscale values.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The drone cruise target detection and tracking method provided in the present application can continue to be locked and tracked after the tracking target is blocked, or after it leaves the field of view and then returns to the field of view, without causing the loss of the tracking target. The specific beneficial effects are described in detail in the specific implementation method.

[0066] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for detecting and tracking a cruise target based on an unmanned aerial vehicle, characterized in that: The following steps are involved: S1: Acquire images, perform target detection, and acquire targets; S2: Tracking and positioning the target; S3: Establishing a database model for the target; S3.1: Obtain the area of ​​the region occupied by the target in the image, obtain a change curve of the area of ​​the region according to the change of the video frame over time, and determine the distance relationship between the target and the photographer according to the change curve; S3.2: Obtain the position of the target in the image, obtain a vector of movement of the position according to changes in the video frame over time, and determine the moving direction of the target according to the vector; S3.3: Obtain the pixel grayscale value of each pixel of the target in the image, divide the grayscale value into multiple levels, and count the proportion of the pixel grayscale value of each level; S3.4: Obtaining data on the proportion of different grayscale values ​​in the area of ​​the target in the image; S3.5: According to the proportion of the grayscale value of each level of pixels and the area of ​​the target in the image, according to the change of the video frame over time, the proportion of different grayscale values ​​​​area analyzes the movement trajectory of the target and the change curve of the proportion of the grayscale value of each level of pixels; S4: Tracking target: When the target is blocked by other objects and re-enters the shooting field of view, the target is tracked according to the moving trajectory of the target and the change curve of the proportion of different gray values; Judging whether the target has moved away and is out of the shooting field of view by the change curve of the area of ​​the region; The vector Z is used to determine whether the target is out of the shooting field of view. When the target re-enters the field of view, the target is tracked according to the reverse direction of the vector Z, the target's motion trajectory, and the change curve of the proportion of different grayscale values.

2. The method for detecting and tracking a target based on a UAV cruise according to claim 1, characterized in that: In S1, the target detection includes using the YOLO algorithm to perform target detection.

3. The method for detecting and tracking a target based on a UAV cruise according to claim 1 or 2, characterized in that: In S2, tracking and locating the acquired target includes adopting the ByteTrack algorithm in the MOT algorithm, and the ByteTrack algorithm is a multi-target tracking algorithm.

4. The method for detecting and tracking a target based on a UAV cruise according to claim 3 is characterized in that: The tracking and positioning of the acquired target also includes a BoT-SORT multi-target tracking algorithm to provide camera motion compensation and improve tracking performance.

5. The method for detecting and tracking a target based on a UAV cruise according to claim 1, characterized in that: The S3.3 step also includes: obtaining a curve of the change in the proportion of pixel grayscale values ​​at each level caused by different backgrounds based on the change of the video frame over time, determining a curve of the change in the range of the pixel grayscale value proportion level of the target, and determining whether the tracked target is the original tracked target based on the change curve.

6. The method for detecting and tracking a target based on a UAV cruise according to claim 1, characterized in that: The step S3.4 also includes: based on the data of the proportion of different grayscale values ​​in the area of ​​the target in the image, the target with the smallest change in grayscale value proportion is identified as the target, which is used to determine whether the tracked target is correct.

7. The method for detecting and tracking a target based on a UAV cruise according to claim 1, characterized in that: The step S3.5 further includes: according to the change curve of the pixel gray value proportion data of each level, when the change curve of the pixel gray value proportion data of the maximum proportion level jumps, it is determined that the target is blocked or out of the shooting field of view.

Images