An image target tracking method for a drone platform

Abstract

The present invention discloses an image target tracking method for a drone platform, based on a DTE-tracker tracking system. The DTE-tracker tracking system includes: a detector, which is composed of the YOLOv5 detection algorithm and is used to identify all moving targets in the video to be tracked; an inspector, which uses the Siamese neural network SiameseNet and is used to inspect all the moving targets to determine the position of the target to be tracked; a tracker, which uses the SiamRPN algorithm and is used to track the target to be tracked. It solves the problem that in the prior art, when facing challenges such as complex backgrounds, target deformation, obstacle occlusion, and interference from similar targets, the drone target tracking effect will deteriorate.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and particularly relates to an image target tracking method for an unmanned aerial vehicle platform. Background Art

[0002] Target tracking refers to predicting the position and size of a target in each frame by using spatial information and temporal correlation within a sequence, and is mainly divided into generative target tracking algorithms and discriminative target tracking algorithms. With the development of neural network technology, discriminative methods based on deep learning have become the mainstream direction of research on target perception, among which the target tracking algorithm based on the Siamese neural network has the most prominent effect. The idea of the Siamese neural network is to use two neural networks with shared weights to map the target template and the search area to the feature space to calculate the similarity, so as to achieve target prediction. In 2016, the target tracking algorithm based on the fully convolutional Siamese network (SiamFC, Fully-Convolutional Siamese Networks for Object Tracking) first applied the Siamese neural network to the target tracking algorithm. SiamFC inputs the target template and the search area into two neural network branches with shared parameters to extract features respectively. Then, the extracted features are subjected to cross-correlation operation to obtain the similarity between different positions of the search area and the target template, and the position with the maximum similarity is the predicted position of the target. In 2018, the target tracking algorithm based on the Region Proposal Network (RPN) (SiamRPN, High Performance Visual Tracking with Siamese Region Proposal Network) integrated the RPN network into the Siamese neural network, and relied on the RPN network to generate candidate regions with different scales and ratios to replace the multi-scale test of the SiamFC algorithm. It is not only more accurate in positioning but also faster in speed, and can process 160 image frames per second. In 2020, the visual target tracking algorithm based on the deep Siamese neural network (SiamRPN++, Evolution of Siamese Visual Tracking with Very Deep Networks) eliminated the center bias by adjusting the method of positive sample sampling during training, and successfully applied the deep network to target tracking.

[0003] The above tracking methods are all short-term target tracking methods, and it is very difficult to re-capture the target after losing it. Currently, in the applied research of UAV target tracking, when there are challenges such as complex background, target deformation, obstacle occlusion, and interference from similar targets, the tracking effect will deteriorate. In the method based on Siamese neural network, although some long-term tracking methods based on full detection have good tracking accuracy, the huge consumption of computing resources makes it almost impossible to be applied to UAVs. Summary of the Invention

[0004] The object of the present invention is to provide an image target tracking method for UAV platforms to solve the problem that in the prior art, when there are challenges such as complex background, target deformation, obstacle occlusion, and interference from similar targets, the UAV target tracking effect will deteriorate.

[0005] The present invention adopts the following technical solutions: An image target tracking method for UAV platforms, based on a DTE-tracker tracking system. The DTE-tracker tracking system includes: a detector, composed of the YOLOv5 detection algorithm, for identifying all moving targets in the video to be tracked; an inspector, using the Siamese neural network SiameseNet, for inspecting all the moving targets to determine the position of the target to be tracked; a tracker, using the SiamRPN algorithm, for tracking the target to be tracked.

[0006] The image target tracking method includes the following steps:

[0007] Step S1: Perform target detection on the video to be tracked through the detector until all moving target sets in the video image to be tracked are detected.

[0008] Step S2: According to the moving target set in Step S1, use the inspector to determine the position of the target to be tracked in the image. The inspector will output the similarity σ between the target to be tracked and the target elements in the moving target set.

[0009] When the similarity σ > σ t That is, when there is a target to be tracked in the moving target set, at this time, the candidate box of the target to be tracked is sent into the tracker for initialization.

[0010] When the similarity σ < σ t That is, when there is no target to be tracked in the moving target set, at this time, return to S1 to restart the detector to search for the target.

[0011] Step S3: Start the tracker to execute the tracking task of the target to be tracked and output the tracking confidence α curve from the SiamRPN algorithm.

[0012] Step S4. Monitor the dynamic target tracking result of the target to be tracked:

[0013] If the target is occluded or interfered, return to step S2 to re-determine the position of the target to be tracked and continue tracking;

[0014] If the target tracking fails, return to step S1 to reactivate the detector to re-capture the target and continue tracking.

[0015] Furthermore, a sliding window method is adopted to monitor the change state of the confidence α curve, and the monitoring mechanism is expressed as:

[0016]

[0017] Among them, let the width of the sliding window be w, and the corresponding frame number when the sliding window starts be f 0 , and its response score is The corresponding frame at the end is f w , and the score is is the minimum value of the confidence in the sliding window frame, The value range of μ is [0, 1].

[0018] Furthermore, the method for judging the target tracking state in step S4 is:

[0019] Let be the maximum value of the confidence in the sliding window frame, that is is the confidence of the corresponding frame number, and i is the frame number sequence number in the sliding window;

[0020] When and μ < μ t , it is considered that the target tracking is normal;

[0021] When and μ > μ t , it is considered that the target tracking is occluded or interfered, return to step S2, and at the same time start the checker;

[0022] When and μ < μ t , it is considered that the target tracking fails, return to step S1, and start the detector.

[0023] The beneficial effects of the present invention are as follows: The present invention is mainly based on the tracking method based on twin neural networks, and at the same time considers the constraints of the UAV itself and the complexity of the environment to study the long-term tracking method applicable to the UAV platform. The DTE-tracker system proposed by an image target tracking method for the UAV platform of the present invention is composed of a detector, an inspector, and a tracker. The system structure is simple and the operation is efficient. Moreover, the network model used by the algorithm is simple, with few parameters and small computational complexity, enabling the system to run in real time on the UAV embedded development board with limited computing resources. An image target tracking method for the UAV platform of the present invention monitors the sliding window in real time, dynamically judges the state of the tracking system, and can make corresponding adjustments, effectively reducing the influence of obstacles, occlusions, similar target interferences, target scale changes, illumination condition changes, etc. existing in the complex environment on the tracking accuracy and stability, so as to ensure that the target tracking can run stably for a long time. Compared with other tracking algorithms, an image target tracking method for the UAV platform designed by the present invention has higher detection and tracking efficiency and more accurate accuracy. It can effectively run on the development board with limited computing power to achieve real-time tracking, and can continuously track specific targets for a long time, improving the robustness of the algorithm in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 FIG. is a schematic diagram of the system framework of a DTE-tracker tracking system of the present invention;

[0025] Figure 2 FIG. is a schematic diagram of the flow of an image target tracking method for the UAV platform of the present invention;

[0026] FIG. 3(a) is a confidence level α curve graph with normal tracking;

[0027] FIG. 3(b) is a confidence level α curve graph with occlusion or interference during tracking;

[0028] FIG. 3(c) is a confidence level α curve graph with tracking failure;

[0029] Figure 4 FIG. is a confidence level α curve graph obtained by a sliding window method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0031] The present invention provides an image target tracking method for the UAV platform, which establishes a dynamic tracking result monitoring and target re-capture mechanism based on a DTE-tracker tracking system, as Figure 1As shown, it mainly consists of a detector, an inspector, and a tracker; the detector is composed of the YOLOv5 detection algorithm, which is used to identify all moving targets in the video to be tracked; the inspector uses the Siamese neural network SiameseNet to check all moving targets and determine the position of the target to be tracked; the tracker uses the SiamRPN algorithm to track the target to be tracked.

[0032] A method for image target tracking on an unmanned aerial vehicle (UAV) platform according to the present invention includes the following steps:

[0033] S1. Read the video to be tracked (or real-time video stream) into the program, start the detector, perform target detection on the video frames, and perform target detection on the video to be tracked through the detector until all moving target sets in the video image to be tracked are detected.

[0034] To meet the conditions of real-time target detection of UAVs and limited onboard resources, the model used by the YOLOv5 detector is n.pt. This model has the simplest network structure and the fastest inference speed in the YOLOv5 series, and is trained after being labeled on a specific dataset collected to meet the requirements of specific target tracking tasks. The set of all moving targets is where are m target boxes in the set of targets to be tracked.

[0035] S2. According to the set of moving targets obtained in step S1, use the inspector to determine the position of the target to be tracked in the image based on the similarity between the image information of the prior target and the elements in the set of moving targets, and send the candidate box of the target to be tracked to the tracker for initialization.

[0036] Specifically, the inspector will output the similarity σ between the target to be tracked and the target elements in the set of moving targets. The threshold σ t is usually [0.70, 0.80];

[0037] When the similarity σ > σ t , that is, there is a target to be tracked in the set of moving targets. At this time, send the candidate box of the target to be tracked to the tracker for initialization;

[0038] When the similarity σ < σ t , that is, there is no target to be tracked in the set of moving targets. At this time, return to S1 and restart the detector to search for targets.

[0039] The inspector in step S2 takes the images of the target to be tracked and the real target as inputs and outputs a similarity score in the range of [0, 1]. The closer the score is to 1, the higher the similarity between the two inputs. The target box T with the highest similarity r =(x r , y r , wr ,h r ) Send it to the tracker to complete the initialization.

[0040] The SiameseNet network used in the present invention is used as an inspector. In order to save computing resources, the ShuffleNetV2 lightweight network is used as the backbone feature extraction network of SiameseNet in the present invention. The SiameseNet network is used as an inspector and the ShuffleNetV2 lightweight network is used as the backbone feature extraction network of SiameseNet. ShuffleNetV2 introduces channel separation operations, pointwise group conv, and bottle-like structures, enhances the interactivity of information, realizes the reuse of features, achieves a high accuracy rate, and greatly improves the efficiency of target feature extraction.

[0041] In this example, a total of 1080 images were collected and manually labeled. These images were taken from the perspectives of drones in outdoor jungle environments under different lighting conditions, multiple viewpoints, different occlusions, different distances, etc. The target part of the person was cut from each image as a new dataset, and then it was combined with the Market1501 pedestrian re-identification dataset to train the SiameseNet inspector.

[0042] S3. After completing the tracker initialization according to step S2, start the tracker to execute the tracking task; and output the tracking confidence α curve from the SiamRPN algorithm.

[0043] S4. Monitor the dynamic target tracking results of the target to be tracked until the tracking task is completed.

[0044] If the target is occluded or interfered, return to execute step S2, re-determine the position of the target to be tracked, and continue tracking;

[0045] If the target tracking fails, return to execute step S1, restart the detector to re-capture the target, and continue tracking.

[0046] In some embodiments, in step S4, dynamic target result monitoring is performed. That is, the output of the SiamRPN tracker is the bounding box and the confidence α, and the α value is a numerical value in the range of [0, 1]. The confidence is monitored in real time using a sliding window method.

[0047] The sliding window method is used to monitor the change state of the confidence α curve, and the monitoring mechanism is expressed as:

[0048]

[0049] Among them, let the width of the sliding window be w, and the corresponding frame number at the start of the sliding window be f 0, whose response score is The corresponding frame at the end is f w , and the score is is the minimum value of the confidence in the sliding window frame, The value range of μ is [0, 1]; according to the actual project engineering experience, the threshold μ t is [0.70, 0.80].

[0050] In some embodiments, let be the maximum value of the confidence in the sliding window frame, that is is the confidence of the corresponding frame number, and i is the frame number sequence number in the sliding window; according to the actual project engineering experience, the threshold α t is 0.75;

[0051] When and μ < μ t , it is considered that the target tracking is normal;

[0052] When and μ > μ t , it is considered that the target tracking has occlusion or interference, return to execute step S2, and start the checker at the same time;

[0053] When and μ < μ t , it is considered that the target tracking fails, return to execute step S1, and start the detector.

[0054] In step S4, there are two cases in the target re-capture mechanism that require re-calling the detector to perform a global target search.

[0055] First, when the target encounters long-term occlusion, the tracking bounding box will stop or drift. When the target appears in the field of view again, the tracker cannot re-capture the target due to its own candidate box selection mechanism;

[0056] When the tracking fails, the confidence α of the bounding box will drop to 0 and remain so for a long time. At this time, it is considered that the target tracking fails, and the checker is started for global search.

[0057] Second, when the tracking result is abnormal, that is, it is necessary to start the checker to determine whether it is a specific target. If the judgment result similarity is less than the set threshold (such as 0.75), it is considered that the target tracking fails, and the checker is started for global search.

[0058] Embodiment

[0059] This example uses an image target tracking method for a drone platform of the present invention. When a human target moves in a jungle environment, due to interference from tree trunks, shrubs, and similar targets, the tracking result may drift to the wrong target, so it is necessary to check the result. Monitoring the change of the confidence α curve can effectively judge the tracking state. Figures 3(a) to 3(c) Shows three states of the tracking result: normal tracking, short-term interference during tracking, and long-term occlusion during tracking. Figures 3(a) to 3(c) The abscissa of is the number of frames of the video, and the ordinate is the tracking response score α.

[0060] When the target is tracked normally, as shown in Fig. 3(a), the α value is close to 1.0. In Fig. 3(b), α fluctuates greatly, drops sharply from 1.0, and then rises to 1. At this time, the tracking result drifts, and it is possible that the tracking box tracks the wrong target, and it is necessary to start the checker to check whether the target is correct. When the target tracking is lost, as shown in Fig. 3(c), the value of α drops from 1 to 0 and then does not change.

[0061] This example uses the method of sliding window to check and monitor the response of the α curve. As Figure 4 shown, the sliding window only slides in one-dimensional space. Let the width of the sliding window be w, and the frame number corresponding to the start of the sliding window be f 0 , and its response score is The frame corresponding to the end is f w , and the score is The monitoring mechanism in Fig. 3(b) can be expressed by Equation (1):

[0062]

[0063] Among them, is the minimum value of the response scores in the sliding window, The value range of μ is [0,1]. When μ exceeds the threshold μ t , that is, short-term occlusion, as shown in Fig. 3(b), the timing to start the checker is: μ > μ t .

[0064] This example adopts a target re-capture mechanism to prevent tracking failure. There are two situations where it is necessary to re-call the detector for global target search.

[0065] First, when the target encounters long-term occlusion, the tracking bounding box will stop or drift. When the target appears in the field of view again, the tracker cannot re-capture the target due to its own candidate box selection mechanism. Similar to the previous text, this embodiment still uses the sliding window method to judge the state of target tracking. As shown in Fig. 3(c), when the tracking fails, the confidence α of the bounding box will drop to 0 and remain so for a long time. Let be the maximum value of the confidence in the sliding window frames, that is, If α t is a threshold value, it is considered that the target tracking fails and detection is initiated for global search.

[0066] Second, when the tracking result fails to pass the checker, that is, the tracked target is incorrect.

[0067] Taking the above two situations into comprehensive consideration, γ is used as the discriminant for calling the detector:

[0068]

[0069] where σ f is the output of the SiameseNet checker, and α t , σ t are the corresponding threshold values respectively. When γ > 0, it means that at least one situation reaches the threshold, that is, the detector is called to recapture the target.

[0070] The present invention designs an image target tracking method for an unmanned aerial vehicle platform. This method can run in real time on an embedded development board of an unmanned aerial vehicle with limited computing resources. This method can effectively reduce the influence of problems such as obstacle occlusion, similar target interference, target scale change, and illumination condition change in a complex environment on the tracking accuracy and stability, so as to ensure that the target tracking can run stably for a long time.

Claims

1. An image target tracking method for an unmanned aerial vehicle platform, characterized in that, based on a DTE-tracker tracking system, the DTE-tracker tracking system includes: a detector, composed of the YOLOv5 detection algorithm, for identifying all moving targets in the video to be tracked; an inspector, using the Siamese neural network SiameseNet, for inspecting all the moving targets to determine the position of the target to be tracked; a tracker, using the SiamRPN algorithm, for tracking the target to be tracked; The image target tracking method includes the following steps: Step S1: Perform target detection on the video to be tracked through the detector until all moving target sets in the video image to be tracked are detected; Step S2: According to the moving target set in Step S1, use the inspector to determine the position of the target to be tracked on the image, and the inspector will output the similarity σ between the target to be tracked and the target elements in the moving target set; When the similarity σ > σ t i.e., there is a target to be tracked in the moving target set, at this time, the candidate box of the target to be tracked is sent to the tracker to complete the initialization; When the similarity σ < σ t That is, the target to be tracked does not exist in the moving target set. At this time, return to S1 to restart the detector to search for the target; Step S3: Activate the tracker to perform the tracking task of the target to be tracked, and output the tracking confidence α curve from the SiamRPN algorithm; Step S4: Monitor the dynamic target tracking result of the target to be tracked: If the target is occluded or interfered, return to execute Step S2 to re-determine the position of the target to be tracked and continue tracking; If the target tracking fails, return to execute Step S1 to reactivate the detector to re-capture the target and continue tracking.

2. The image target tracking method for an unmanned aerial vehicle platform according to claim 1, characterized in that, the method of sliding window is adopted to monitor the change state of the confidence α curve, and the monitoring mechanism is expressed as: Among them, let the width of the sliding window be \(w\), and when the sliding window starts, the corresponding frame number is \(f\). 0 , and its response score is When it ends, the corresponding frame is \(f\). w , and the score is is the minimum value of the confidence in the sliding window frames, The value range of \(\mu\) is \([0, 1]\).

3. The image target tracking method for an unmanned aerial vehicle platform according to claim 2, characterized in that, the method for judging the target tracking state in Step S4 is: Let be the maximum value of the confidence in the sliding window frame, that is be the confidence corresponding to the frame number, and i is the frame number sequence number in the sliding window; When and μ < μ t , it is considered that the target tracking is normal; When and μ > μ t , it is considered that the target tracking is occluded or interfered, return to execute step S2, and at the same time start the checker; When and μ < μ t , it is considered that the target tracking fails, and step S1 is returned for execution to start the detector.

Images