Target tracking method based on particle filtering and deep distance metric learning

Abstract

The present invention discloses a target tracking method based on particle filtering and deep distance metric learning, which relates to the field of autonomous driving target visual tracking. It includes the following steps: constructing a nonlinear deep metric learning model; based on a given set of positive and negative samples of autonomous driving targets, training the nonlinear deep metric learning model and optimizing its nonlinear deep learning model parameters based on the gradient descent method; constructing a target observation model based on particle filtering to obtain the optimal estimate of the state of the autonomous driving target; updating the target template through an online tracking strategy that combines short-term and long-term stable updates to achieve effective tracking of autonomous driving targets. The proposed method performs well in scenes such as partial occlusion and illumination changes; compared with the comparison algorithm, in most test scenarios, the proposed method has a lower average center error and a higher average overlap rate, indicating that the overall tracking performance of the proposed method is better.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving target visual tracking, and in particular to an autonomous driving target visual tracking method based on particle filtering and deep distance metric learning. Background Art

[0002] Autonomous driving involves many fields such as information perception, information processing, and decision-making. Among them, information perception, as the basic module for collecting driving environment information, involves many information collection sensors such as lidar, millimeter-wave radar, ultrasonic radar, GPS, and cameras. As a sensor that can collect rich scene information and is inexpensive, cameras have been regarded by the industry as standard scene information perception devices for autonomous driving. As a result, camera-based autonomous driving target tracking has become one of the research hotspots in the field of computer vision. In recent years, many efficient and robust autonomous driving visual tracking algorithms have been proposed, which has greatly promoted the practical application of target visual tracking. However, due to the complexity of actual autonomous driving scenes, there are a lot of interference and uncertain factors such as illumination changes, size changes, and target occlusion in the tracking process, which leads to a significant decrease in tracking performance. Therefore, how to improve the accuracy and robustness of autonomous driving target visual tracking algorithms in complex scenes is still one of the research difficulties in the field of visual tracking.

[0003] In order to solve the problem of decreased performance of target visual tracking in complex scenes, Nam H et al. proposed a deep learning tracking method. This method first trained the network offline and then fine-tuned the network parameters to finally obtain a relatively good network model. However, due to the problems of long time consumption and poor feature training pertinence, Zhang KH et al. proposed a visual tracking algorithm using convolutional network (convolutional network tracker, CNT). This method first used the k-means algorithm to construct a feature atlas, then used an adaptive threshold shrinkage algorithm to reduce noise on the training result image, and finally constructed a target model based on sparse representation. However, the convolution operation of this algorithm will reduce the resolution of the feature extraction map. To solve the above problems, Lu XK et al. proposed a regression network that maps samples to a labeled response map. However, due to the possible dimensional mismatch between the target and the background, the author proposed a method that considers the loss function of shrinkage loss and performs regression. Hu J proposed to improve the problem of distinguishing targets from backgrounds by using stacked independent subspace analysis networks to learn nonlinear distance metrics. The discriminative deep metric learning (DDML) method uses a large margin criterion at the top layer of the trained deep network to explicitly obtain the nonlinear distance metric. However, since a very large auxiliary dataset is required and the auxiliary dataset may not be consistent with the objects captured online, the learned features cannot adapt to these objects. Summary of the invention

[0004] In view of the performance degradation of traditional target tracking methods in complex environments, the present invention proposes a target tracking method based on particle filtering and deep distance metric learning, comprising the following steps:

[0005] Build nonlinear deep metric learning models;

[0006] Based on a given set of positive and negative samples of an autonomous driving target, training the nonlinear deep metric learning model and optimizing its nonlinear deep learning model parameters based on a gradient descent method;

[0007] Build a target observation model based on particle filtering to obtain the optimal estimate of the target state of autonomous driving;

[0008] An online tracking strategy that combines short-term and long-term stable updates is used to update the target template, thereby achieving effective tracking of autonomous driving targets.

[0009] Due to the adoption of the above technical scheme, the present invention can achieve the following technical effects: the autonomous driving target visual tracking method proposed by the present invention that integrates deep distance metric learning and particle filtering has high target tracking accuracy and robustness for target tracking in complex environments. The proposed method constructs a nonlinear deep metric learning model based on a deep network; then optimizes the parameters of the obtained deep metric learning model based on a gradient descent algorithm; then constructs an observation model based on the obtained optimal candidate target prediction value to obtain the optimal estimate of the autonomous driving target state; finally, the target template is updated based on an update strategy that combines short-term and long-term stable updates to achieve effective tracking of autonomous driving targets. From the qualitative analysis, it can be seen that the proposed method performs better in scenes such as partial occlusion and illumination changes; based on the quantitative analysis, it can be seen that compared with the comparison algorithm, in most test scenarios, the average center error of the proposed method is lower and the average overlap rate is higher, indicating that the overall tracking performance of the proposed method is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 Implementation flow chart of the present invention;

[0012] Figure 2 Tracking result diagram for five different tracking algorithms;

[0013] Figure 3 Tracking success rate curves for different tracking methods;

[0014] Figure 4 Tracking overall accuracy curves for different tracking methods. DETAILED DESCRIPTION

[0015] The implementation steps of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments: In view of the problem that the target tracking performance is significantly reduced due to factors such as illumination changes, target deformation, partial occlusion, etc. in complex environments, the present invention proposes a method for visual tracking of targets in autonomous driving based on particle filtering and deep distance metric learning. The proposed method first learns the hierarchical nonlinear transformation in the feedforward neural network based on the deep metric learning method; then, the template and the particles are mapped to the same feature space, and the intra-class difference of the positive training pairs and the inter-class variation of the negative training pairs in the feature space are minimized; then, based on the particle filter framework, the candidate most similar to the template obtained by the deep metric network is identified as the real target; finally, the template is updated online based on an update strategy that combines short-term and long-term stable updates to reduce the impact of adverse factors and achieve effective tracking of autonomous driving targets. Experimental results show that compared with the existing mainstream tracking algorithms, the proposed method has higher target tracking accuracy and better robustness in complex environments. The specific steps include the following:

[0016] Step 1. Build a nonlinear deep metric learning model, specifically:

[0017] Build a deep network model by Pass it to multiple layers of nonlinear transformation to learn its nonlinear representation. The number of network layers k = 1, 2, ..., K, r (k) Represents the sample dimension in the k-th layer network, then the first layer sample output is expressed as follows:

[0018]

[0019] Among them, W (1) represents the projection matrix of the first layer network, b (1) is the bias of the first layer network. φ is the S-type nonlinear activation function.

[0020] The output of the first layer network can be used as the input of the second layer network and recursively, then the output of the kth layer network can be expressed as:

[0021]

[0022] in,

[0023] sample The output of the top layer of the network can be expressed as:

[0024]

[0025] Among them, the mapping f is a parameter nonlinear function, which is composed of parameters and Joint decision.

[0026] Based on the above method, the sample x is represented by the constructed deep network model i With x j The Euclidean distance between them is used to measure the similarity between the two:

[0027]

[0028] To learn parameters in the deep network model and Based on the marginal Fisher analysis criterion, the following nonlinear deep metric learning function is constructed:

[0029]

[0030] in, Represents sample x i With x j Belong to the right, Represents sample x i With x j Belongs to negative pairs. α is a positive parameter that can weigh the internal compactness of positive sample pairs and the separability of samples within negative sample pairs, and β (β>0) is a regularization parameter. represents the Frobenius norm of the matrix, M and N represent the total number of positive and negative pairs in the training data, respectively.

[0031] Step 2: Based on a given set of positive and negative samples of the autonomous driving target, the nonlinear deep metric learning model is trained and its nonlinear deep learning model parameters are optimized based on the gradient descent method.

[0032] Given a training sample set x=(x1,x2,...,x n )∈R and sample positive and negative sample pairs based on the zero-mean Gaussian distribution around the target sample image block, where the six-parameter diagonal covariance matrix of the positive sample for autonomous driving is diag([1,1,0,0,0,0]), which means sampling within a radius of two pixels around the target; the six-parameter diagonal covariance matrix of the negative sample for autonomous driving is diag([w,h,0,0,0,0]), where w and h represent the width and height of the target sample, respectively, and sampling is performed from distant negative samples far away from the target object. Therefore, some negative samples may contain both the background and part of the body of the target object. Based on the obtained training set, randomly form M positive pairs and N negative pairs and then train the parameters of the deep network;

[0033] Since the constructed nonlinear depth metric model is non-convex, it is difficult to directly obtain a closed-form solution. In order to solve the above optimization problem, a gradient descent method is used to solve the parameter W (k) With b (k) , specifically:

[0034]

[0035]

[0036] in, represents the original input sample, then the sample x i x j Step length and The top level of can be expressed as follows:

[0037]

[0038]

[0039] in, Representation function About W (K) The derivative of . Other layer variables and It is expressed as follows:

[0040]

[0041]

[0042] Among them, ⊙ represents element-by-element multiplication, It can be expressed as follows:

[0043]

[0044] Update the parameter W based on the gradient descent algorithm (k) With b (k) , k = 1, 2, ..., K until convergence:

[0045]

[0046]

[0047] Among them, η is the learning rate, which is used to control the convergence speed of the objective function L.

[0048] Step 3. Build a target observation model based on particle filtering to obtain the optimal estimate of the target state of autonomous driving, specifically:

[0049] Assume that the target state vector of autonomous driving at time r is h r ={h rx ,h ry ,sc r ,θ r ,φ r ,σ r}, where h rx ,h ry ,sc r,θ r ,φ r ,σ r is the six-degree-of-freedom affine transformation parameter, and the target motion model between adjacent frames can be expressed as follows:

[0050]

[0051] in, express The mean is h r-1 , a Gaussian distribution with variance Σ, where Σ is the diagonal covariance matrix.

[0052] Since candidate targets are estimated only in the nearest frame, the motion model When fixed, the optimal candidate target can be directly based on the observation model Select, that is:

[0053]

[0054] Among them, Γ is the normalization factor, γ is the constant that controls the shape of the Gaussian kernel, and is taken as 0.01 in the simulation.

[0055] Step 4. Update the target template through an online tracking strategy that combines short-term and long-term stable updates to achieve effective tracking of the autonomous driving target. Specifically:

[0056] In the actual tracking process, keeping the target template unchanged cannot effectively track the changing targets in complex scenes, so template updating has always been a hot issue in online target tracking. If tracking is implemented based on a fixed template from the first frame, factors such as illumination changes, background clutter or partial occlusion will cause the tracker to be unable to capture the target well; conversely, if the template is updated quickly, each update will introduce errors, and as time goes by, the errors gradually accumulate, causing the tracker to deviate from the target. In response to the above problems, the present invention introduces an online tracking strategy that combines short-term and long-term stable updates to update the target template.

[0057] Template initialization: First, determine the location of the target in the first frame, then obtain the tracking results of the first n frames based on the proposed tracking method and normalize them, and finally combine them into a template set T = [t1, t2, ···, t n ]∈R b×n ;

[0058] Template dynamic update: The similarity between the template and the tracking result can be expressed as ψ=[ψ1,ψ2,···,ψ n ], let the threshold be ρ, then the similarity between the tracking result and the i-th template is ψ i It is expressed as:

[0059]

[0060] In the formula, is the tracking result of the rth frame, and the similarity value ψ i The larger the value, the more similar the tracking result is to the template.

[0061] Assume the maximum similarity is Λ, which is expressed as:

[0062] Λ=maxψ i

[0063] Compare the maximum similarity Λ with the threshold ρ. If the maximum similarity Λ>ρ, it means that the current tracking result has the greatest similarity with a target template, and the corresponding template is updated; otherwise, no update is made. In the simulation experiment, the threshold value is ρ=0.7.

[0064] The effect of the present invention can be further illustrated by the following simulation:

[0065] Simulation conditions: The hardware environment is as follows: Intel Core(TM) i5-4258 CPU, main frequency 2.4GHz, memory 8GB, experimental software test environment: Python3.7, MATLAB 2017a and open source deep learning framework Caffe. The simulation conditions are set as follows: the number of positive and negative samples extracted by the proposed tracking algorithm in the first frame is 100 and 400 respectively, and the number of positive and negative samples in subsequent frames is 30 and 120 respectively, generating 300 positive pairs and 900 negative pairs. Weighing the tracking performance and computational complexity, if there are too many particles, the algorithm calculation amount will increase, and if there are too few particles, the optimal target state cannot be obtained. Based on this, the number of particles per frame is set to 600, and the particle weight is initialized to 1 / 600. The video tracking dataset OTB-100, a total of 6 video sequences including MotorRolling, Boy, Skating1, Bird2, Tiger2, and Basketball, which contain multiple tracking challenge factors, are selected as the test set. The CNN network used in the present invention adopts the deep learning framework Caffe, and the network weight update adopts the gradient descent method. The local area normalization parameters α=0.0001, τ=0.75 are set to play a "side inhibition" role and enhance the generalization ability of the network to extract complex environmental information; the learning rate is set to 0.001, and the training cycle is 300 to minimize the occurrence of "overfitting". Among them, the present invention uses the average tracking overlap rate and the average center position error to quantitatively analyze the tracking performance of the proposed method.

[0066] Simulation content:

[0067] Simulation 1: Qualitative analysis: Figure 2The results of five tracking algorithms for six test sequences are compared. The MotorRolling sequence contains challenging factors such as rapid motion, background clutter and illumination changes. In the 112th frame, the target changes significantly from falling from the air. MIL and BACF have tracking drift or the tracking frame does not match the real target. However, the proposed algorithm can always track the target well, which can be attributed to the fact that the proposed algorithm considers the influence of background clutter and rapid motion to accurately estimate the moving target. In Basketball, the target has obvious size changes. The proposed algorithm and BCAF can locate the target and track it effectively, indicating that the proposed method also has good tracking effect under the condition of size change. In Boy, the target moves rapidly, and there are interferences such as scale change and rotation. MIL has tracking drift after 418 frames. Skating1 is a more complex scene with low contrast between the target and the background and strong illumination changes. In this scene, the target resolution is low. The proposed algorithm updates the template in time by combining long and short time online update strategy, thereby achieving stable tracking. The proposed algorithm can lock the target well in Bird2 video sequence and Tiger2 video sequence.

[0068] Simulation 2: Quantitative analysis: As shown in Tables 1 and 2, based on the six test sequences selected by OTB-100, the proposed algorithm has better tracking effects than the comparison algorithms. This can be attributed to the fact that the proposed algorithm uses deep distance metric learning and introduces error terms to construct a likelihood model to reduce the sensitivity between similar target backgrounds. Compared with the comparison algorithm, the proposed algorithm performs well under conditions such as occlusion and noise. The main reasons can be described as follows:

[0069] (1) The proposed model considers the correlation between candidate target templates, which improves the robustness of algorithm tracking in complex scenarios;

[0070] (2) Deep distance metric measures particle similarity, thereby improving tracking effectiveness;

[0071] (3) The long-short time update strategy improves the robustness and tracking accuracy of the proposed algorithm in noisy and occlusion scenarios.

[0072] Table 1 Average overlap rate of different tracking methods

[0073]

[0074] Table 2 Average center position error of different tracking methods

[0075]

[0076] The present invention uses the success rate curve and the overall accuracy graph to evaluate the overall performance of the tracker. The overall accuracy graph represents the percentage of successful frames with center position error within the distance threshold to the total number of frames. The success rate and overall accuracy curves obtained by the comparison algorithm are shown in Figure 2. Figure 3 ,4 As shown. Figure 3 , 4 It can be seen that the tracking success rate of the proposed algorithm in most sequences is higher than that of the comparison algorithm; the tracking accuracy curve of the proposed algorithm in the Tiger2 sequence is slightly lower than that of BCAF, but the tracking success rate curve is still better than BCAF, and the overall tracking accuracy of the proposed algorithm in other sequences is also better than that of the comparison algorithm. It can be seen that the overall performance of the proposed algorithm in complex scenes is better than that of the comparison method, and it has good robustness.

[0077] Simulation 3: Average running speed of different tracking methods in each test sequence: To verify the tracking timeliness of the proposed algorithm, the present invention uses the number of frames per second (FPS) to measure the algorithm speed (the algorithm runs 50 times, and the average FPS is used as the evaluation index). The FPS obtained by each algorithm in different test sequences is shown in Table 3. As can be seen from Table 3, the speed of the proposed algorithm is higher than Struck, BCAF and DFT, but lower than MIT. However, as mentioned above, the tracking performance of the proposed algorithm in each test sequence is better than the comparison algorithm as a whole.

[0078] Table 3 Average running speed (FPS) of different tracking methods under each test sequence

[0079]

[0080] In summary, the present invention proposes a method for visual tracking of autonomous driving targets based on particle filtering and deep distance metric learning. The proposed method constructs a nonlinear deep distance metric learning model based on a deep network; then constructs an observation model based on the obtained optimal candidate target prediction value; finally, the target template is updated based on an update strategy combining short-term and long-term stable updates. Based on the six test sequences selected from the OTB-100 data set containing factors such as occlusion and illumination changes, the effectiveness of the proposed method is verified by comparing with four mainstream trackers such as BACF, MIL, Struck and DFT. From the qualitative analysis, it can be seen that the proposed method performs better in scenes such as partial occlusion and illumination changes; based on the quantitative analysis, it can be seen that compared with the comparison algorithm, in most test scenes, the average center error of the proposed algorithm is lower and the average overlap rate is higher, indicating that the overall tracking performance of the proposed method is better. Therefore, the algorithm proposed in the present invention can provide a solid theoretical and engineering implementation basis for visual tracking of autonomous driving targets in complex driving scenarios.

[0081] The embodiments of the present invention have better practicability and are not intended to limit the present invention in any form. The technical features or combinations of technical features described in the embodiments of the present invention should not be considered isolated, and they can be combined with each other to achieve better technical effects. The scope of the preferred embodiments of the present invention may also include other implementations, and this should be understood by those skilled in the art of the invention embodiments.

Claims

1. A target tracking method based on particle filtering and deep distance metric learning, characterized in that: include: Build nonlinear deep metric learning models; Based on a given set of positive and negative samples of an autonomous driving target, training the nonlinear deep metric learning model and optimizing its nonlinear deep learning model parameters based on a gradient descent method; Build a target observation model based on particle filtering to obtain the optimal estimate of the target state of autonomous driving; An online tracking strategy combining short-term and long-term stable updates is used to update the target template, thereby achieving effective tracking of autonomous driving targets. The positive and negative samples of autonomous driving targets are obtained by using the nonlinear changes of the deep network, and then a nonlinear deep metric learning model is constructed. Specifically: Build a deep network model by Pass it to multiple layers of nonlinear transformation to learn its nonlinear representation; the number of network layers k = 1, 2, ..., K, r (k) Represents the sample dimension in the k-th layer network, then the first layer sample output is expressed as follows: Among them, W (1) represents the projection matrix of the first layer network, b (1) is the deviation of the first layer network; φ is the S-type nonlinear activation function; The output of the first layer network is used as the input of the second layer network and recursively. The output of the kth layer network is expressed as: in, sample The output of the top layer of the network is expressed as: Among them, the mapping f is a parameter nonlinear function, which is composed of parameters and Joint decision making; Based on the above method, the sample x is represented by the constructed deep network model i With x j The Euclidean distance between them is used to measure the similarity between the two: To learn parameters in the deep network model and Based on the marginal Fisher analysis criterion, the following nonlinear deep metric learning function is constructed: in, Represents sample x i With x j Belong to the right, Represents sample x i With x j belongs to a negative pair; α is a positive parameter that weighs the internal compactness of the autonomous driving target positive sample pair and the internal sample separability of the autonomous driving target negative sample pair; β (β>0) is a regularization parameter; represents the Frobenius norm of the matrix, M and N represent the total number of positive and negative pairs in the training data, respectively; Based on a given set of positive and negative samples of autonomous driving targets, the nonlinear deep metric learning model is trained and its nonlinear deep learning model parameters are optimized based on the gradient descent method, specifically: Given a training sample set x=(x1,x2,...,x n )∈R and sample positive and negative sample pairs based on the zero-mean Gaussian distribution around the target sample image block, where the six-parameter diagonal covariance matrix of the autonomous driving target positive sample is diag([1,1,0,0,0,0]), which means sampling within a radius of two pixels around the target; the six-parameter diagonal covariance matrix of the autonomous driving target negative sample is diag([w,h,0,0,0,0]), where w and h represent the width and height of the target sample, respectively, and sampling is performed from negative samples far away from the target object; based on the obtained training set, randomly form M positive pairs and N negative pairs and then train the parameters of the deep network; Use the gradient descent algorithm to solve for the parameter W (k) With b (k) , specifically: in, represents the original input sample, For vector The transpose of , then the sample x i x j Step length and The top-level representation of is as follows: in, Representation function About W (K) Derivatives of; other layer variables and It is expressed as follows: Among them, ⊙ represents element-by-element multiplication, for: Update the parameter W based on the gradient descent algorithm (k) With b (k) , k=1,2,...,K until convergence: Among them, η is the learning rate, which is used to control the convergence speed of the objective function L; The target observation model is constructed based on particle filtering to obtain the optimal estimation of the target state of autonomous driving. Specifically, assuming that the target state vector of autonomous driving at time r is h r ={h rx ,h ry ,sc r ,θ r ,φ r ,σ r }, where h rx ,h ry ,sc r ,θ r ,φ r ,σ r is the six-degree-of-freedom affine transformation parameter, and the driving target motion model between adjacent frames is expressed as follows: in, express The mean is h r-1 , Gaussian distribution with variance Σ, Σ is the diagonal covariance matrix; motion model When fixed, the optimal candidate target is directly based on the observation model Select, that is: Among them, Γ is the normalization factor, γ is the constant that controls the shape of the Gaussian kernel; The online tracking strategy that combines short-term and long-term stable updates is used to update the target template to achieve effective tracking of the autonomous driving target. Specifically: First, the position of the first frame of the driving target is determined, and then the tracking results of the first n frames are obtained and normalized based on the proposed tracking method, and finally they are combined into a template set T = [t1, t2, …, t n ]∈R b×n ; The similarity between the template and the tracking result is expressed as ψ=[ψ1,ψ2,···,ψ n ], let the threshold be ρ, then the similarity between the tracking result and the i-th template is ψ i It is expressed as: In the formula, is the tracking result of the rth frame, and the similarity value ψ i The larger the value, the more similar the tracking result is to the template; Assume the maximum similarity is Λ, which is expressed as: L = maxψ i The maximum similarity Λ is compared with the threshold ρ. If the maximum similarity Λ>ρ, it indicates that the current tracking result has the greatest similarity with a certain target template, and the corresponding template is updated; otherwise, it is not updated.

Images