Siamese network target tracking method combining template updating and trajectory prediction

Abstract

The present invention discloses a twin network target tracking method combining template updating and trajectory prediction, including: obtaining a target tracking model through offline training of a preprocessed data set, extracting image features of the current frame through the target tracking model, and comparing it with the template image to determine the position information of the target to be tracked in the next frame in the image; during the entire target tracking process, the template image is updated online, and the template image features are continuously and iteratively updated; at the same time, according to the target motion position change information in the historical frames, the Kalman filtering method is used to predict the trajectory to achieve stable tracking of a single target.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking methods, and relates to a twin network target tracking method combining template updating and trajectory prediction. Background Art

[0002] As an important branch of computer vision, target tracking has been increasingly widely used in autonomous driving, human-computer interaction, intelligent transportation, and other fields. Target tracking technology aims to obtain relevant parameters such as the real-time position, motion trajectory, motion speed, and acceleration of the target through the tracking process, and conduct advanced visual task analysis based on this, so as to achieve an understanding of the target's behavior. Existing target tracking methods can be divided into generative methods and discriminative methods according to different observation models. Generative methods only focus on the target itself, while discriminative methods also focus on the background area around the target for targeted training. Therefore, discriminative methods can better handle scenes with more complex environments around the target. With the development of deep learning in recent years, discriminative target tracking algorithms based on deep learning have achieved remarkable results. The target tracking algorithm based on the twin network structure is a typical discriminative target tracking method based on deep learning. The twin network refers to a coupled architecture based on two artificial neural networks. The fully convolutional Siamese network (SiamFC) proposed by Bertinetto et al. is a pioneering work that uses a dual-branch twin network structure to deal with target tracking problems. It uses two fully convolutional networks with exactly the same structure to extract the features of the target area and the search area, and calculates the similarity between the two. The position with the highest similarity is used as the predicted target position. A large number of excellent frameworks have been optimized and improved based on this structure. Valmadre et al. proposed an asymmetric twin network that combines neural networks with correlation filters. Guo et al. proposed a dynamic twin network that learns to track the appearance changes of the target online and performs background suppression. Choi et al. proposed an attention-related filter network that introduced the attention mechanism into the tracking framework. Yuan et al. proposed an anti-occlusion tracking algorithm that extracts target features from different layers of the residual neural network (Residual Neural Network, ResNet) to generate a response map, and introduced an occlusion detection strategy to avoid model drift.

[0003] However, the above target tracking algorithms still have the following shortcomings, which limit the performance of the algorithms: 1) The twin network often uses offline training, does not make full use of online tracking information, and these methods always use the target selected in the first frame as the template image. When the target is deformed, the color or lighting changes, it cannot identify the target well, and lacks the update iteration of the template image. 2) The twin network locates the target through the cross-correlation of the template image and the search image, does not use the target trajectory information during the tracking process, and cannot stably track fast-moving or occluded targets. Summary of the invention

[0004] The purpose of the present invention is to provide a twin network target tracking method combining template updating and trajectory prediction, which solves the problem of poor target tracking stability in the prior art.

[0005] The technical solution adopted by the present invention is a twin network target tracking method combining template updating and trajectory prediction, including:

[0006] Step 1: Establish a target tracking model, which includes a feature extraction branch, a positive and negative sample classification branch, and a position information regression branch;

[0007] Step 2: Select the target bounding box of the initial frame image x0, crop the template image at the center of the target bounding box of the initial frame image x0, and extract the features of the template image to obtain the features of the template image;

[0008] Step 3: Initialize the Kalman filter, input the center position of the target bounding box of the initial frame image x0 into the Kalman filter, and predict the next frame image x t,t＝1…n Target center location;

[0009] Step 4: Get the next frame image x t , according to the previous frame image x t-1 The center position of the target bounding box and the current frame image x t The target center position is fused to produce the current frame image x t The target search center position is obtained by cropping the search image at the target search center position, and feature extraction is performed on the search image to obtain search image features;

[0010] Step 5: Perform cross-correlation operation on the template image features and the search image features to obtain a response map; input the response map into the target tracking model to obtain the current frame image x t The classification map and regression map are used to smooth the target motion using the cosine window and scale change penalty. Then, according to the position with the highest score in the classification map, the corresponding regression position information is selected in the regression map to obtain the current frame image x t Predicted object bounding box;

[0011] Step 6: Determine whether the template image needs to be updated based on the response graph; if necessary, update the template image in the current frame image x t The template image is cropped at the center of the predicted target bounding box, and the template image features are extracted and updated; otherwise, no update is performed;

[0012] Step 7: Get the current frame image x t The predicted target bounding box information is converted into the position and scale information of the image, input into the Kalman filter, and the next frame image x is predicted. t+1 The trajectory position of the target in the image and determine the next frame image x t+1 Target center position; if tracking is completed, end tracking; otherwise proceed to the next step;

[0013] Step 8: Let t=t+1, and repeat steps 4-7 until the tracking is completed.

[0014] The present invention is also characterized in that:

[0015] The process of establishing the target tracking model in step 1 includes:

[0016] Obtain the training data set, preprocess the training data set, input the preprocessed training data set into the target tracking model, perform classification training of positive and negative samples, and regression training of position information respectively. The loss function of the target tracking model is L = L cls +L reg ;

[0017] Among them, the classification loss function

[0018] In the above formula, N is the total number of samples, y i is the label of sample i, positive sample is 1, negative sample is 0, p i The probability of predicting sample i as a positive sample;

[0019] The regression loss function is defined as L reg = 1-IoU;

[0020] In the above formula, |A∩B| represents the intersection of the predicted target bounding box and the actual target bounding box, and |A∪B| represents the union of the predicted target bounding box and the actual target bounding box.

[0021] The preprocessing method is: crop the images in the training data set and perform data enhancement processing at the same time.

[0022] In step 6, the response graph is judged by APCE:

[0023]

[0024] In the above formula, F max 、F min and F w,h They represent the maximum value, minimum value and value at the (w,h) coordinate in the response graph respectively.

[0025] The formula for updating the template image features in step 6 is:

[0026]

[0027] In the above formula, η is the update step size of the template image, β1 and β2 are the parameters for judging APCE and F respectively. max The ratio threshold for whether to update, APCE avg , They are the historical average APCE value, the historical average F max value.

[0028] The beneficial effects of the present invention are as follows: the twin network target tracking method combining template updating and trajectory prediction of the present invention, by performing real-time online updates and iterations on the template image during the tracking process, always pays attention to changes in the shape and appearance of the target, and continuously maintains the effectiveness of the template image, thereby improving the target tracking performance of the existing twin network-based target tracking method when the target changes shape, color or illumination; the Kalman trajectory prediction method is introduced in the target tracking process, the target position information is collected during the tracking process and the trajectory is predicted, and the target center point is predicted to determine the area to be searched, thereby effectively utilizing the trajectory information of the target during the tracking process, further solving the problem of the target being obscured or moving rapidly, and improving the target tracking accuracy and precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a reasoning flow chart of the twin network target tracking method combining template updating and trajectory prediction of the present invention;

[0030] Figure 2 It is a framework diagram of the twin network target tracking method combining template updating and trajectory prediction of the present invention;

[0031] Figure 3 It is a training flow chart of a feature extraction model in a twin network target tracking method combining template updating and trajectory prediction of the present invention;

[0032] Figure 4 It is an EAO comparison diagram of the twin network target tracking method combining template updating and trajectory prediction of the present invention and other tracking methods;

[0033] Figure 5 It is a radar chart of the tracking effect of the twin network target tracking method combining template updating and trajectory prediction of the present invention and other tracking methods. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The twin network target tracking method that combines template updating and trajectory prediction, such as Figure 1 and Figure 2 As shown, the specific steps include:

[0036] Step 1: Figure 3 As shown in FIG. 1 , a training data set is obtained, and the image size in the training data set is uniformly cropped to a size of 511×511. Then, data enhancement processing is performed on the training data set, including random small shifts and scaling of the target length and width. A target tracking model is established, and the target tracking model includes a feature extraction branch, a positive and negative sample classification branch, and a position information regression branch. The image pairs sampled from the video or still image in the preprocessed training data set are input into the target tracking model, and classification training of positive and negative samples and regression training of position information are performed respectively. The parameters in the model are optimized using the stochastic gradient descent algorithm (SGD) to minimize the total loss value. The loss function of the target tracking model is L=L cls +L reg ;

[0037] The classification training process of positive and negative samples is as follows: the ratio of positive samples to negative samples is set to 1:3, and the classification loss function is

[0038] In the above formula, N is the total number of samples, y i is the label of sample i, positive sample is 1, negative sample is 0, p i The probability of predicting sample i as a positive sample;

[0039] The regression loss function is defined as L reg = 1-IoU;

[0040] In the above formula, |A∩B| represents the intersection of the predicted target bounding box and the actual target bounding box, and |A∪B| represents the union of the predicted target bounding box and the actual target bounding box.

[0041] Step 2: Figure 2 As shown, the target bounding box of the initial frame image x0 is selected, and a 127×127 image block is cropped at the center of the target bounding box of the initial frame image x0 as a template image, and the target tracking model is input to extract the features of the template image to obtain the features of the template image;

[0042] Step 3: Initialize the Kalman filter, input the center position of the target bounding box of the initial frame image x0 into the Kalman filter, and predict the next frame image x t,t＝1…n Target center location;

[0043] Step 4: Get the next frame image x t , according to the previous frame image x t-1 The center position of the target bounding box and the current frame image x t The target center position is fused to produce the current frame image x t The target search center position is determined, a search image of size 255×255 is cropped at the target search center position, and the search image feature is extracted by inputting the target tracking model to obtain the search image feature;

[0044] Step 5: Perform cross-correlation operation on the template image features and the search image features to obtain a response map; input the response map into the target tracking model to obtain the current frame image x t The classification map and regression map are used to smooth the target motion using the cosine window and scale change penalty. Then, according to the position with the highest score in the classification map, the corresponding regression position information is selected in the regression map to predict the current frame image x t Target bounding box;

[0045] Step 6: Use APCE (average correlation peak energy) to determine whether the template image needs to be updated. If necessary, the current frame image x obtained in step 5 is updated. t The image block of size 127×127 is cropped at the center of the target bounding box as the template image, and the template image features are extracted. The template image features y are updated according to the formula t ; Otherwise, no update is performed;

[0046]

[0047] In the above formula, F max 、F min and F w,h They represent the maximum value, minimum value and value at (w,h) coordinates in the response graph respectively;

[0048] The formula for updating the template image features is:

[0049]

[0050] In the above formula, η is the update step size of the template image, β1 and β2 are the parameters for judging APCE and F respectively. max The ratio threshold for whether to update, APCE avg , They are the historical average APCE value, the historical average F maxValue; In each frame image, if the APCE value of the response graph and F max If the values ​​of all exceed the historical mean under a certain proportion, it is considered that the target in the current frame is in a good state, and the tracking result of the current frame should be learned to update the template image to adapt to the gradual change of the target during the tracking process. Otherwise, no update is performed to prevent contamination of the template image.

[0051] Step 7: Get the current frame image x t The target bounding box information is converted into the position and scale information of the image, input into the Kalman filter, and the next frame image x is predicted t+1 The trajectory position of the target in the image and determine the next frame image x t+1 Target center position; if tracking is completed, end tracking; otherwise proceed to the next step;

[0052] Step 8: Let t=t+1, and repeat steps 4-7 until the tracking is completed.

[0053] In this embodiment, the Kalman trajectory prediction method is introduced in the target tracking process, and the position information obtained by the twin network tracker is used as the observation value of the Kalman filter, so that the prediction result of the Kalman filter is more stable, which makes up for the disadvantage that the twin network tracker can only track offline and does not have an online learning process, and further realizes the combination of offline tracking and online learning, and effectively utilizes the trajectory information of the target during the tracking process. The Kalman filter includes two processes: time update and observation update. Time update is mainly used for prediction of the tracking system, including state prediction and covariance prediction. Observation update is mainly used for correction of the tracking system, including covariance update, state update and Kalman gain update. The main idea of ​​the Kalman filtering method is to use the observation value to correct the prediction value, so as to obtain a more accurate tracking bounding box. The first is the prediction of the state:

[0054]

[0055] in, It represents the prior state estimate at time k, which is the intermediate calculation result of the filter, that is, the result of time k predicted based on the optimal estimate at the previous time (k-1), which is the result of the prediction equation. represents the posterior state estimate at time k-1, which is the updated result, i.e. the optimal estimate. k-1 is the system input. A and B are the state transfer matrix and the matrix that converts the input to the state, respectively. Then the covariance corresponding to the system result is predicted:

[0056]

[0057] in, represents the prior estimated covariance at time k, P k-1represents the posterior estimated covariance at time k-1, Q is the covariance matrix of the system process noise, and represents the error between the state transition matrix and the actual process. The results obtained from the previous state prediction can be used to update the state observation. First, the Kalman gain is calculated:

[0058]

[0059] Among them, K k is the filter gain matrix, i.e., the Kalman gain. H is the observation matrix, and R is the measurement noise covariance matrix. The optimal state estimate at time k can be calculated by the observation value, predicted value, and Kalman gain of the current state.

[0060]

[0061] in, and They represent the posterior state estimate and the prior state estimate at time k, respectively, and z k Represents the observation value at time k, which is the input of the filter. Finally, the covariance matrix corresponding to the state is updated:

[0062]

[0063] Among them, P k and They represent the posterior estimated covariance and the prior estimated covariance at time k respectively, and I is the unit matrix.

[0064] Through the above methods, the twin network target tracking method combining template updating and trajectory prediction of the present invention, by performing real-time online updates and iterations on the template image during the tracking process, always pays attention to changes in the shape and appearance of the target, and continuously maintains the effectiveness of the template image, thereby improving the target tracking performance of the existing twin network-based target tracking method when the target changes shape, color or illumination. The Kalman trajectory prediction method is introduced in the target tracking process to collect the target position information during the tracking process and perform trajectory prediction. By predicting the target center point, the area to be searched is determined, and the trajectory information of the target during the tracking process is effectively utilized, further solving the problem of the target being obscured or moving rapidly, and improving the target tracking accuracy and precision.

[0065] like Figure 4 , Figure 5As shown, the average overlap expected effect diagram and the tracking effect radar diagram under different circumstances of evaluating the twin network target tracking method combining template update and trajectory prediction on the VOT2018 dataset, the VOT2018 dataset includes 60 video test sequences with different challenges, and its main evaluation index is the average overlap expectation (Expected Average Overlap, EAO), which comprehensively considers the accuracy (Accuracy, A) and robustness (Robustness, R). The method of the present invention is compared with ATOM, DiMP, Ocean, SiamDW, SiamMask, SiamR-CNN, and SiamRPN++ methods. From the results in the figure, it can be seen that the single target tracking method of the present invention is superior to other methods, and effectively improves the performance of the single target tracking method of the algorithm under the conditions of target occlusion, target deformation, camera movement, and target rapid movement.

Claims

1. A twin network target tracking method combining template updating and trajectory prediction, characterized in that: The following steps are involved: Step 1: Establish a target tracking model, wherein the target tracking model includes a feature extraction branch, a positive and negative sample classification branch, and a position information regression branch; Step 2: select the target bounding box of the initial frame image x0, crop the template image at the center of the target bounding box of the initial frame image x0, and extract the features of the template image to obtain the features of the template image; Step 3: Initialize the Kalman filter, input the center position of the target bounding box of the initial frame image x0 into the Kalman filter, and predict the next frame image x t,t＝1…n Target center location; Step 4: Get the next frame image x t , according to the previous frame image x t-1 The center position of the target bounding box and the current frame image x t The target center position is fused to produce the current frame image x t a target search center position, cropping at the target search center position to obtain a search image, and performing feature extraction on the search image to obtain search image features; Step 5: perform cross-correlation operation on the template image features and the search image features to obtain a response map; input the response map into the target tracking model to obtain the current frame image x t The classification map and regression map are used to smooth the target motion using the cosine window and scale change penalty. Then, according to the position with the highest score in the classification map, the corresponding regression position information is selected in the regression map to predict the current frame image x t Target bounding box; Step 6: judging whether it is necessary to update the template image according to the response graph; If necessary, in the current frame image x t The template image is cropped at the center of the target bounding box, and the template image features are extracted and updated; Otherwise, no update is performed; Step 7: Get the current frame image x t The target bounding box information is converted into the position and scale information of the image, input into the Kalman filter, and the next frame image x is predicted t+1 The trajectory position of the target in the image and determine the next frame image x t+1 Target center position; if tracking is completed, end tracking; otherwise proceed to the next step; Step 8: Let t=t+1, and repeat steps 4-7 until the tracking is completed.

2. The twin network target tracking method combining template updating and trajectory prediction according to claim 1 is characterized in that: The process of establishing the target tracking model in step 1 includes: Obtain a training data set, preprocess the training data set, input the preprocessed training data set into the target tracking model, perform classification training of positive and negative samples, and regression training of position information respectively. The loss function of the target tracking model is L = L cls +L reg ; Among them, the classification loss function In the above formula, N is the total number of samples, y i is the label of sample i, positive sample is 1, negative sample is 0, p i The probability of predicting sample i as a positive sample; The regression loss function is defined as L reg = 1-IoU; In the above formula, |A∩B| represents the intersection of the predicted target bounding box and the actual target bounding box, and |A∪B| represents the union of the predicted target bounding box and the actual target bounding box.

3. The twin network target tracking method combining template updating and trajectory prediction according to claim 2 is characterized in that: The preprocessing method is: cropping the images in the training data set and performing data enhancement processing at the same time.

4. The twin network target tracking method combining template updating and trajectory prediction according to claim 1 is characterized in that: In step 6, the response graph is judged by APCE: In the above formula, F max 、F min and F w,h They represent the maximum value, minimum value and value at the (w,h) coordinate in the response graph respectively.

5. The twin network target tracking method combining template updating and trajectory prediction according to claim 1 is characterized in that: The formula for updating the template image features in step 6 is: In the above formula, η is the update step size of the template image, β1 and β2 are the parameters for judging APCE and F respectively. max The ratio threshold for whether to update, APCE avg , They are the historical average APCE value, the historical average F max value.

Images