UAV target tracking method based on local image data learning

Abstract

A target tracking method based on learning from local image data includes: densely sampling the area around the target, where the sampled areas are considered potential targets, and areas outside the sampling frame are discarded; extracting local features of the image based on the local data while simultaneously learning the local data; using a structural support vector machine as a classifier and employing a discriminant tracking method; calculating the saliency features of the image to obtain the position of the object; fusing the tracked position obtained by the structural support vector machine with the object position obtained by the saliency method to determine the final object position; and updating the positive and negative support vectors based on the predictions. The method provided by the present invention can achieve good tracking results in various scenarios (such as occlusion, light changes, and scale changes).

Description

Technical Field

[0001] The present invention relates to a method for implementing target tracking by utilizing computer pattern recognition technology, and in particular to a method for unmanned aerial vehicle target tracking based on local image data learning. Background Art

[0002] In traditional video object tracking, classic tracking algorithms such as subspace learning, sparse representation, multi-task learning, and multi-instance learning all use various features to model the target. These algorithms then track the target by comparing it with the target or using a learned classifier to classify the potential target. These algorithms have achieved some success on common object tracking databases. Tracking methods can generally be categorized into generative and discriminative approaches.

[0003] The generative tracking method is to model the potential target and then compare it with the existing target through some metric. The closest one is considered to be the target. Earlier video target tracking used the color histogram of the target as a feature and compared the potential target with the target template through the Bhattacharyya coefficient (Condensation—conditional density propagation for visual tracking, IJCV, 29(1), 5-28, 1998). The subspace method performs PCA decomposition on the target and obtains the principal component of the target as a feature for tracking (Incremental learning for robust visual tracking, IJCV, 77, 125-141, 2008). Superpixel features are also used as target features in target tracking (Superpixel tracking, ICCV, 2011). After sparse representation was introduced into computer vision, it was also used for target tracking (Robust visual tracking using l1minimization, ICCV, 2009). Here, the pixel information of the target is directly used as the target feature, and the target is found by comparing the distance between the potential target and the template. The paper Robust visual tracking via multi-task sparse learning in [IEEE conference on computer vision and pattern recognition, 1-8, 2012] mentions the introduction of multi-task learning methods into target tracking.

[0004] The above target tracking methods are all based on the idea of error minimization, while the discriminant method uses the data in the image to train a classifier, and uses the classifier to classify the targets in the image, thereby finding the foreground target. The multi-instance learning algorithm calculates the Harr features of the target, and then uses the multi-instance algorithm to classify the potential targets (Visual tracking with online multiple instance learning, in CVPR, 2009). The literature (Support vector tracking, TPAMI, 26 (8), 1064-1072, 2004) uses the support vector machine (Support Vector Machine, SVM) as a classifier. The literature (Structured output tracking with kernels. In ICCV, 263-270, 2011) uses the structure support vector machine (SSVM) for classification. The structure support vector machine classifier has achieved good results in target tracking experiments due to its fast speed and high accuracy.

[0005] However, currently known tracking methods mostly track general targets, such as people and cars. In standard videos, people and cars are relatively large and are seen at eye level. Such images have been studied for a long time in computer vision, so existing features can well describe these objects. In target tracking, features significantly influence tracking effectiveness. For example, the aforementioned tracking methods utilize image histograms, pixel intensity, Haar features, and superpixel features. However, when tracking drones, targets are very small due to their long distances, and only their outlines can be seen. Therefore, features are required that can accurately capture the target's edge information. Furthermore, the target scene varies greatly, which requires features to have a certain degree of learning capability. Summary of the Invention

[0006] In order to solve the defects of the existing technology, the present invention proposes a feature based on local data learning, combined with a structured support vector machine classifier, adopts a discriminant tracking method, and uses the target's saliency information to assist tracking.

[0007] A target tracking method based on local image data learning includes learning based on local image data to obtain local features, and then implementing target tracking in combination with a fusion video of salient features under the framework of discriminant tracking.

[0008] Another target tracking method based on local image data learning includes the following steps: (1) dense sampling around the target, the sampled objects are potential targets, and those outside the sampling frame are discarded; (2) based on the local data, local features of the image are extracted, and the local data are learned at the same time to improve the stability to target changes and the robustness to image noise and geometric deformation; (3) using the structural support vector machine as a classifier, adopting the discriminant tracking method, improving the extracted local features to have good classification ability, and overcoming the problem of tracking loss caused by poor feature selection in the previous structural support vector machine classifier; (4) calculating the saliency features of the image to obtain the position of the object; (5) through the tracking position of the structural support vector machine and the object position obtained by saliency, the two positions are fused to determine the final position of the object. (6) based on the prediction, the positive and negative support vectors are updated.

[0009] Therefore, the local features selected by this method have good classification ability and are not very sensitive to changes in lighting conditions, posture, etc., and are robust to slight physical noise and geometric deformation.

[0010] Another technical solution provided by the present invention is to realize drone video tracking by combining it with a structured support vector machine, which achieved a high accuracy and success rate in drone tracking experiments.

[0011] A target tracking method based on local image data learning is applied to UAV video tracking. The method includes the following steps:

[0012] The first step is to use the particle filter method to sample the area near the target's previous frame position to obtain potential targets.

[0013] In the second step, the features of each potential target are calculated based on the potential targets obtained in the first step.

[0014] The third step is to use the structural support vector machine to classify the targets and find the best target.

[0015] The fourth step is to calculate the feature saliency around the object.

[0016] In the fifth step, the saliency information obtained in the first step and the target position obtained in the third step are used to fuse the target position and determine the final target position.

[0017] In the sixth step, sampling is performed at the final position of the target to obtain positive and negative samples for training, and the model of the structural support vector machine is updated online.

[0018] This paper proposes a local feature that is highly capable of describing target edge information and exhibits a certain degree of robustness against noise and geometric deformation. It then combines this feature with a structured support vector machine (SSVM) to design a discriminative tracking method, and uses target saliency information to assist in tracking. The proposed tracking method achieves excellent tracking results in various scenarios, such as occlusion, light changes, and scale variations. It can be applied to various civilian and military systems, including face tracking, drone tracking, and military target tracking systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart of an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of geodesic distance.

[0021] Figure 3 Schematic diagram of significant information.

[0022] Figure 4 Schematic diagram showing the experimental results of several methods on the drone database.

[0023] Figure 5 Schematic diagram showing the experimental results of the success rates of several methods on the drone database.

[0024] Figure 6 Schematic diagram showing the experimental results of the accuracy of several methods on the drone database (scale change scenario).

[0025] Figure 7 Schematic diagram showing the experimental results of the accuracy of several methods on the drone database (scenes with varying aspect ratios).

[0026] Figure 8 Schematic diagram showing the experimental results of several methods on the UAV database (low-resolution scene).

[0027] Figure 9 Schematic diagram showing the experimental results of the success rates of several methods on the drone database (scale change scenario).

[0028] Figure 10 Schematic diagram showing the experimental results of the success rate of several methods on the drone database (scenes with varying aspect ratios).

[0029] Figure 11 Schematic diagram showing the experimental results of the success rates of several methods on the drone database (low-resolution scene).

[0030] Figure 12 The figure shows the tracking results of bicycle videos on the UAV database using the method of the present invention.

[0031] Figure 13 It shows the tracking results of the method of the present invention on the ship video in the UAV database.

[0032] Figure 14 The figure shows the tracking results of the shipboard video in the UAV database using the method of the present invention.

[0033] Figure 15 It shows the tracking results of the crowd video on the drone database using the method of the present invention. DETAILED DESCRIPTION

[0034] The following is a detailed description of an embodiment of the present invention in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0035] like Figure 1 As shown, the objectives of this embodiment are firstly to enable the features of the present invention to effectively describe the target and secondly to enable the features to overcome noise interference. The approach taken is to use geodesic distance on the manifold instead of Euclidean distance and to consider the image gradient. SSVM is used for classification to determine the target location. The target's saliency information is then considered and fused to achieve final target location.

[0036] The technical solution of the present invention involves geodesic distance, local features, structural support vector machine method, and calculation of significance method, and is described in detail as follows.

[0037] The geodesic distance involved in the present invention is described as follows:

[0038] Figure 2 The difference between geodesic distance and Euclidean distance is given. Assume that on a surface, the data can be represented as {x1,x2,z(x1,x2)}. Where x1 and x2 are coordinates, and z(x1,x2) represents how the surface changes with x1 and x2. That is, the surface can be represented as S(x1,x2) = {x1,x2,z(x1,x2)}. Then the length of the curve in the figure can be expressed as:

[0039] The local features of the present invention are described as follows:

[0040] If we expand the above formula for calculating the length of the curve, we can get:

[0041]

[0042] where Δx = [dx1, dx2] T , Since we use a local operator here, +ΔxT Δx is a very small variable and can be ignored, so ds 2 ≈Δx T CΔx. Therefore, our local feature can be expressed as: K(C, Δx) = exp(-ds 2 )=exp{-Δx T CΔx}, where C is calculated as follows:

[0043]

[0044] Where u1 and u2 represent the eigenvectors of matrix C, s1 and s2 represent the eigenvalues of matrix C, and the values of ∈, τ, and α are selected as 10 -8 , 0.4 and 0.2.

[0045] The structural support vector machine method of the present invention is described as follows:

[0046] It is also used to handle multi-classification problems. The advantage of support vector machines lies in their strong theoretical foundation, namely their strong generalization ability. However, their disadvantages are 1) high training complexity and 2) inability to predict structured problems. Structural support vector machines extend support vector machines from binary classification problems to structured prediction problems by modifying their constraints and objective function.

[0047] The goal of structural support vector machine training is to learn the prediction function f. In the following formula, g is the scoring function. The scoring function outputs a continuous value instead of the traditional [+1,-1]. y is a rectangle in the search space, t is the current frame, time space, and the function f is to map the time space to the coordinate space, that is, the function of the position prediction process

[0048] f(t)=argmax y∈Y g(t,y)

[0049] Using a series of sample training to solve the following problem is the loss function of the basic support vector machine, which belongs to the maximum margin framework, minimizing w, with a soft margin term:

[0050]

[0051]

[0052] The Δ symbol represents the interval we defined. The actual meaning of this interval is related to the overlap rate. When the overlap rate is 1, the value is 0, which is equivalent to the sample at the center during training, with the interval set to 0.

[0053]

[0054] Represents the two tracking positions y and degree of overlap.

[0055] The calculation of the significant features involved in the present invention is explained as follows:

[0056] Image saliency is an important visual feature in an image, reflecting the degree of attention that the human eye pays to certain areas of the image. Since the work of Itti (L. Itti, C. Koch, & E. Niebur. A model of saliency based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11): 1254-1259, 1998.) in 1998, a large number of saliency mapping methods have been developed, and image saliency has also been widely used in image compression, coding, image edge and region enhancement, salient object segmentation and extraction, etc. For an image, the user is only interested in some areas of the image. These areas of interest represent the user's query intent, while the majority of the remaining uninterested areas are irrelevant to the user's query intent. Salient areas are the areas in the image that are most likely to arouse the user's interest and best represent the image content. In fact, the selection of salient areas is very subjective. Due to different user tasks and knowledge backgrounds, different users may choose different areas as salient areas for the same image. A common method is to calculate the saliency of an image based on the human attention mechanism. Research in cognitive psychology shows that certain areas in images significantly attract attention and contain a large amount of information. Cognitive scientists have proposed numerous mathematical models to simulate human attention mechanisms. By leveraging general principles of image cognition, the extracted salient areas are more consistent with human subjective evaluation. In this patent, we use image saliency to infer the location of an object.

[0057] After obtaining the local eigenvalues of the image, we use the Matrix Cosine Similarity (MCS) to calculate the similarity between the features at each position and the features at other positions. The saliency is calculated as follows:

[0058]

[0059] Among them F i and F j Represent the local features at positions i and j respectively.

[0060] The present invention fuses the tracking position of the structural support vector machine and the object position obtained by saliency to determine the final object position. The calculation method for the target position fusion is as follows:

[0061] y=α1y1+α2y2

[0062] Among them, y is the final target, y1 and y2 are the target positions obtained by using the structural support vector machine and the saliency features, respectively, and α1 and α2 are the weights.

[0063] On the basis of the above technologies, the technical solution of the present invention is specifically implemented as follows:

[0064] 1) Sampling to obtain potential targets

[0065] Dense sampling is performed around the target, and the sampled objects are potential targets, and those outside the sampling box will be discarded.

[0066] 2) Calculate local features.

[0067] For each potential object in the image, we compute local features.

[0068] 3) Use structural support vector machine classification to obtain the target position.

[0069] Use support vector machine to output scores for the location of each potential target and predict the location of the target in the current frame.

[0070] 4) Calculate significance.

[0071] A box area is determined near the target of the previous frame obtained by tracking, and the saliency of the image is calculated.

[0072] 5) Position fusion.

[0073] The target positions obtained in step 3) and step 4) are fused and calculated as follows:

[0074] y=α1y1+α2y2,

[0075] Among them, y is the final target, y1 and y2 are the target positions obtained in step 3) and step 4), and α1 and α2 are weights.

[0076] 6) Online update.

[0077] Based on the predictions, the positive and negative support vectors are updated, and then the structural support vector machine model is updated.

[0078] The experimental data uses a drone database sourced from: https: / / ivul.kaust.edu.sa / Pages / Dataset-UAV123.aspx. This dataset provides 123 videos from 12 different scenes. The longest video has 3085 frames, with a total of over 110,000 frames. The tracking scenarios are quite diverse, including roads, areas near buildings, and grasslands. The tracked targets include people, cars, and boats. The 12 scenes are:

[0079] Aspect ratio change: The aspect ratio of the object changes beyond the range of [0.5, 2].

[0080] Background Clutter: The background and target are very similar.

[0081] Camera Motion: The camera shakes suddenly.

[0082] Fast Motion: The object moves very quickly between the previous and next frames.

[0083] Full Occlusion: The target is completely occluded.

[0084] Illumination Variation: The brightness of light on the surface of an object changes dramatically.

[0085] Low Resolution: The size of the object is less than 400 pixels.

[0086] Out-of-View: Part of the object is out of the camera range.

[0087] Partial Occlusion: Part of the object is blocked.

[0088] Similar Object: There are other similar objects near the target.

[0089] Scale Variation: The scale variation of an object exceeds the range of [0.5, 2].

[0090] Viewpoint Change: The angle at which you observe an object changes significantly.

[0091] At the same time, the algorithms compared with the method of this embodiment include: kernel correlation filtering tracking method (shown as "Comparison Algorithm 1" in the figure, which is a sparser dotted line), detection learning tracking algorithm (shown as "Comparison Algorithm 2" in the figure, which is a denser dotted line) and support vector machine-based tracking algorithm (shown as "Comparison Algorithm 3" in the figure, which is a long and short dotted line).

[0092] We used two common evaluation criteria in computer vision: precision score and success score. Precision represents the difference between the center of the tracked position and the center of the ground-truth position. Success score represents the overlap rate between the tracked position box and the ground-truth position box.

[0093] Figure 4 The accuracy of different methods on the drone database is shown in Figure 1. In the figure, "Our Algorithm" represents the tracking method of our invention, represented by a solid line. It can be seen that our method achieves a relatively good accuracy on the database drones.

[0094] Figure 5 It shows the success rate of different methods on the UAV database. It can be seen that the present invention has achieved a better success rate on the database UAV.

[0095] Figure 6-Figure 8 The figure shows the accuracy of different methods in various scenarios on the drone database, with a total of 12 scenarios. It can be seen that the present invention achieves better accuracy in each scenario.

[0096] Figures 9-11 The success rates of different methods in various scenarios on the drone database are shown in Table 2. It can be seen that the present invention achieves a better success rate in various scenarios.

[0097] Figure 12 The figure shows the tracking results of bicycle videos on the UAV database using the method of the present invention.

[0098] Figure 13 It shows the tracking results of the method of the present invention on the ship video in the UAV database.

[0099] Figure 14 The figure shows the tracking results of the shipboard video on the UAV database using the method of the present invention.

[0100] Figure 15 The figure shows the tracking results of the method of the present invention on the crowd video in the drone database.

Claims

1. A target tracking method based on local image data learning, characterized by This involves learning local features based on local image data, and then implementing target tracking by combining fusion videos with salient features within the framework of discriminative tracking. The specific steps are as follows: (1) Dense sampling is performed around the target. The sampled objects are potential targets, and those outside the sampling frame are discarded; (2) Extracting local features of the image based on local data and learning the local data at the same time; (3) Using structural support vector machines as classifiers and adopting discriminant tracking methods to improve the extracted local features; (4) Calculate the salient features of the image to obtain the location of the object; (5) The tracking position of the structural support vector machine and the object position obtained by saliency are fused to determine the final object position; (6) Update the positive and negative support vectors based on the prediction; The local features are expressed as: K(C,Δx)=exp(-ds 2 )=exp{-Δx T CΔx}, where Δx = [dx1, dx2] T ,ds 2 ≈Δx T CΔx; Among them, C is calculated as follows: Where u1 and u2 represent the eigenvectors of matrix C, s1 and s2 represent the eigenvalues of matrix C, and the values of ε, τ, and α are selected as 10 respectively. -8 , 0.4 and 0.

2.

2. The target tracking method based on local image data learning according to claim 1, characterized in that The goal of the step (3) structural support vector machine training is to learn the prediction function f, where g is the scoring function. The scoring function outputs a continuous value instead of the traditional [+1,-1]. y is a rectangle in the search space, t is the current frame, time space, and the function f is to map the time space to the coordinate space, that is, the function of the position prediction process. f(t)=argmaxy∈Yg(t,y) Using a series of sample training to solve the following problem is the loss function of the basic support vector machine, which belongs to the maximum margin framework, minimizing w, with a soft margin term: The Δ symbol represents the interval we defined. The actual meaning of this interval is related to the overlap rate. When the overlap rate is 1, the value is 0, which is equivalent to the sample at the center during training, with the interval set to 0. Represents the two tracking positions y and degree of overlap.

3. The target tracking method based on local image data learning according to claim 1, characterized in that The saliency of the computational image is calculated as follows: Where Fi and Fj represent the local features at positions i and j respectively; y is a rectangle in the search space and t is the current frame.

4. The target tracking method based on local image data learning according to claim 1, characterized in that The tracking position obtained by the structural support vector machine and the object position obtained by saliency are fused to determine the final object position. The calculation method for fusion of the target position is as follows: y=α1y1+α2y2 Among them, y is the final target, y1 and y2 are the target positions obtained by using the structural support vector machine and the saliency features, respectively, and α1 and α2 are the weights.

Images