A method of twin network target tracking based on incremental learning

Abstract

The application discloses a kind of based on incremental learning's twinborn network target tracking method, incremental learning is applied to the updating process of target tracking network model;First, the RPN of tracking network SiamRPN++ is copied as student model, and the high-confidence target generated in the tracking process is used as the small sample set of online training;Then, the small sample set generated by the last frame network is learned using the incremental learning mode, and the student model is trained by domain expansion and knowledge distillation;Finally, the target information generated by the student network model and the target information generated by the teacher network model are dynamically weighted and fused to update the position;The application uses the incremental learning mode to make the offline trained model have self-adaptive learning ability, not only effectively utilizes the historical information of the target in the tracking process, but also avoids large-scale offline training of the model, improves the ability of the twinborn network algorithm to handle target deformation in the tracking process.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of based on incremental learning's twin network target tracking method, belong to computer vision's target tracking technical field. BACKGROUND

[0002] Target tracking refers to the initial frame specified target in video sequence, and the motion state of the target is continuously predicted in subsequent video frames. The motion state of the target object usually refers to the position and size of the target object in the new video frame, which is represented by a rectangular box.

[0003] The tracking algorithm at present stage can be roughly divided into two categories. One applies the idea of correlation filter to the tracking field, mainly divided into kernel cycle structure algorithm and kernel correlation filter algorithm. This kind of algorithm converts the solution of the tracker template from time domain complex operation to Fourier domain point multiplication calculation, greatly reduces the amount of calculation, and the speed of tracker is significantly improved, but the accuracy is not ideal. Correlation filter algorithm uses the features extracted by neural network instead of manually labeled features, which can improve the accuracy to a certain extent, but the calculation efficiency is greatly reduced during model updating. The other category is the tracking algorithm represented by deep learning. The recently developed twin network tracking algorithm has good performance in accuracy and speed. SiamFC, a full convolutional twin network, is the pioneer of this kind of algorithm. SiamFC obtains the feature extraction ability of the object through large-scale offline training, and then calculates the similarity between the search area and the template picture during tracking. The position with the highest response is the estimated position of the target. For the SiamFC framework, a series of improved algorithms are proposed, including the introduction of complementary twin network branch, attention mechanism, graph convolutional neural network, and the use of reinforcement learning to adjust model parameters. For example, Li et al. proposed SiamRPN, which introduced RPN structure into SiamFC. Through the classification branch, the position of the target is obtained, and through the regression branch, the accurate estimation of the size of the target is obtained. After that, DaSiamRPN improves the discrimination ability of the twin network by mining negative samples during the training stage. Li et al. introduced ResNet network into the twin network and proposed SiamRPN++ algorithm, which achieved the best performance on multiple target tracking datasets. VOIGTLAENDE et al. proposed Siam R-CNN, a twin region convolutional neural network, which uses re-detection to improve the success rate of tracking. UpdateNet uses the initial frame, historical frame and current frame to update the template, fully utilizes the appearance information of target changes, and improves the robustness of tracking process. DiMP algorithm adds samples through data augmentation, learns the convolutional network of online update branch through gradient descent, and intermittently updates the model to further improve the comprehensive performance of the algorithm. Ocean uses anchor free and uses fast conjugate algorithm to train online branch during inference process, which fully explores the accuracy and success rate in the direction of tracking segmentation.

[0004] Although the field of target tracking has developed rapidly in recent years, the challenges of the tracking problem make target tracking still a difficult task. In the tracking video, due to the target object constantly undergoing deformation, occlusion, rotation, scale change and illumination transformation, etc., the tracking task is extremely challenging. In the tracking video sequence, the target object is usually determined by the initial frame. The twin network generates a target template using the initial frame, and then completes target tracking by template matching in the subsequent frame. In the actual scene, if the tracker only uses the target template of the initial frame, it may cause the target to be lost in subsequent tracking, because the target will change during movement. At present, there are three main ways to adjust the model using the historical information of the target: fixed update, adaptive update and online learning. Fixed update, i.e. linear superposition of target template in tracking process, the update rate of update process is constant, but the update effect of this method is not good, and it lacks flexibility. Adaptive update, i.e. training a separate network to learn the appearance change trend of the target, such as the update network of UpdateNet needs additional offline training to learn the motion changes of various objects, but this method is complex and has many training stages, and can only be trained for a single test dataset, lacking the generalization ability of other scenes. Online learning, i.e. using the tracking result of the previous moment to train a separate feature extraction network online, and using this network for inference at the next moment. Such as DiMP, a separate small network is trained during model inference, usually the high confidence target tracked is selected as the sample, and gradient descent is used to make the small network learn, and the network inference speed is guaranteed by indirect training, but this method will forget the old knowledge during training, and cannot fully utilize the already trained content. In view of the above problems of lack of flexibility, complex training and forgetting old knowledge, it is necessary to design an effective online adjustment model tracking method. SUMMARY

[0005] In view of the different shortcomings in the prior art described above, the present application aims to provide a twin network target tracking method based on incremental learning: the teacher-student network is used to expand the domain of the tracking target, and the feature change loss is used to reduce the difference between the initial target template and the new target tracking result, thereby improving the network's ability to classify the foreground and background of the new target; the distillation loss is used to prevent the model from forgetting the old knowledge during training, which can enhance the ability to continuously utilize the historical appearance feature information of the target, thereby solving the deformation problem existing in the target tracking process, and improving the applicability of the model in different scenes.

[0006] In order to achieve the above object, the technical scheme adopted by the present application is to provide a twin network target tracking method based on incremental learning, through domain expansion training and distillation loss during online training of the student network, so that the model can learn new appearance information during the adjustment process without forgetting the past features. Its features mainly include:

[0007] 1. Based on the target tracking model, the target tracking model includes a twin network tracking module, a sample generation module, a template linear update module, an online training module, a domain expansion module, a knowledge distillation module and a selection module, wherein the functions of each module are:

[0008] The twin network tracking module is used for similarity calculation between the target template and the search area in the current video frame, the input is the target template and the search area in the current video frame, the initial value of the target template is set by manual frame selection in the first frame, and the output is the tracking bounding box of the target in the next frame.

[0009] The sample library generation module is used to center on the high confidence target, randomly move the tracking box, and make the intersection over union (IOU) with the tracking target greater than a certain threshold as a positive sample, generally set greater than 0.8, and the IOU less than a certain threshold as a negative sample, generally set 0.1, the positive and negative proportion of the positive and negative sample library is a dynamic value, and is a certain quantity, the tracking response size obtained by the algorithm determines, that is, when high confidence tracking result is generated, the positive sample proportion and quantity are increased; when low confidence tracking result is generated, the negative sample proportion and quantity are increased. When the new result in the next frame is generated, the sample library is replaced;

[0010] The template linear update module is used for linear weighting of the features of the tracked target in the initial frame and the features of the high confidence samples after subsequent tracking;

[0011] The online training module comprises a domain expansion module and a knowledge distillation module. The online training is used to copy the RPN network of SiamRPN++ as a student network at the initial frame, and the original RPN network is frozen as a teacher network to guide the student network learning. The small sample library is converted into labeled image data, and the online training of the student network is performed, and the cross-entropy loss is used as the loss function to optimize the classification branch in the student network; and the feature loss function and the distillation loss function are added to the domain expansion and knowledge distillation branches respectively, and the target is continuously tracked after training, and this module is a prerequisite for the domain expansion module and the knowledge distillation module. The domain expansion module is used to freeze some layers of the classification branch of the student network during the training process to maintain the decision boundary. The feature change loss is used to reduce the difference in feature mapping between the initial target template and the new target tracking result; the knowledge distillation module is used to maintain the prediction ability of the student network to the past appearance information of the target. In order to introduce the distillation loss into the old model as additional supervision, after the teacher network and the student network are simultaneously input with the same frame of image and the results are predicted respectively, the prediction result of the teacher network is used to guide the training process to prevent catastrophic forgetting.

[0012] The selection module is used to complete the subsequent calculation operation of the twin network tracking box, and the classification branches of the student network and the teacher network are fused, and the fusion method can adopt linear weighting.

[0013] 2. The twin tracking module comprises two branches and three region proposal network units (RPN), each branch being a feature extraction unit, the first branch being used to extract the features of the target template, and the input being the target template, and the second branch being used to extract the features of the search region, and the input being the search region; each branch extracts three kinds of features, which are shallow, medium and deep layer features; the three region proposal network units are in a cascade structure, and the output of the cascade fusion is subjected to window penalty and the like, and the target bounding box of the search region is output, wherein the fusion proportion constant of each region proposal network is obtained by training; the corresponding inputs of the first, second and third region proposal network units are the shallow, medium and deep layer features of the target template and the search region.

[0014] 3. The working process of the online training module is as follows:

[0015] (1) At the initial frame, the RPN network of the original model is copied as a student network, and the original model RPN network is frozen as a teacher network;

[0016] (2) The results of the twin network tracking are used to generate a positive and negative sample library, the positive samples are classified as targets, and the negative samples are classified as backgrounds, which are sent to the student network for normal training, and only one cycle of training is performed, and the learning rate is consistent with the original algorithm setting;

[0017] (3) The training process adds a domain expansion module, which uses feature change loss to reduce the difference in feature mapping between the initial target template and the new target tracking result.

[0018] 4. The working process of the domain expansion module is as follows:

[0019] (1) After the current positive and negative sample library is extracted by the backbone network feature, it is respectively passed through the deep related convolution part of the classification and regression branch of the teacher and student network, to obtain the feature before inputting Head;

[0020] (2) The features before inputting Head obtained by the two networks and the features extracted by the backbone network are calculated by mean square error, and feature change loss is used to reduce the difference in feature mapping between the initial target template and the new target tracking result. The feature loss function is as follows:

[0021]

[0022] In the formula, x is the feature map of the image extracted by the backbone network; and are the weights of the teacher model and the student model, respectively; represents the feature before the classifier in RPN; F(·) is mean square error (MSE); and a is a hyperparameter used to balance the loss.

[0023] (3) After the training is completed, the student model parameters are saved, and the next frame of image is entered, while the teacher and student networks are used for normal tracking.

[0024] 5. The working process of the distillation module is as follows:

[0025] (1) When a new video frame is sent into the twin network tracking module for tracking, the student model after online training of the previous frame is still used for related calculation, and the credibility of the tracking results of the teacher network and the student network is compared;

[0026] (2) If the credibility of the teacher network is greater than that of the student network, the distillation loss training is performed, and if the credibility of the teacher network is less than or equal to that of the student network, the distillation loss training is not performed. The prediction results of the teacher network and the student network are recorded as p T and p S , respectively.

[0027]

[0028] In the formula, F(·) is a loss function, which aims to reduce the difference between the two distributions, and β is a hyperparameter used to balance the loss. p S’ is a part of p S , and y' T is the prediction of the student on the last frame information of the tracked target. y' Sis p T and p S transformations, called "soft labels". The converter is a modified softmax function, where:

[0029]

[0030] where T is a smoothing parameter, called temperature. The higher the temperature, the softer the labels, i.e. the flatter the probability distribution.

[0031] (3) After the distillation loss training is performed, the classification responses of the teacher network and the student network are weighted and sent into the selection module to complete the final regression of the tracking frame and the like.

[0032] The beneficial effects of the present application are:

[0033] The domain expansion learning method of the teacher-student network designed by the present application is to use the feature change loss to reduce the difference between the feature mapping of the initial target template and the new target tracking result, thereby improving the foreground and background classification ability of the network on the new target. In the knowledge distillation module, the distillation loss is used to prevent the model from forgetting the old knowledge after training, which can enhance the continuous use of the historical appearance feature information of the target, thereby solving the problems of deformation and occlusion in the target tracking process, and the general ability of the model in different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is the overall structure schematic diagram of the twin network target tracking method based on incremental learning provided by the present application;

[0035] Figure 2 is the flow chart of each module of the twin network target tracking method based on incremental learning provided by the present application;

[0036] Figure 3 is the structure schematic diagram of the teacher-student network provided by the present application;

[0037] Figure 4 is the structure schematic diagram of the distillation loss provided by the present application; DETAILED DESCRIPTION

[0038] The present application will be further described below in combination with the drawings and examples.

[0039] The present application provides a twin network target tracking method based on incremental learning, which is described below with reference to Figure 1 and Figure 2 The method comprises the following steps:

[0040] Step 1: constructing a SiamRPN++ network framework;

[0041] The first branch is used for extracting features of the target template, and the input is the target template; the second branch is used for extracting features of the search area, and the input is the search area; each branch extracts three kinds of features, which are shallow, medium and deep features, corresponding to the SiamRPN++ structure, that is, the feature outputs of the second layer, the third layer and the fourth layer of the ResNet50 backbone network; the three region proposal network units are in a cascaded structure, and the output of the three units after cascaded fusion is subjected to regression operations such as window penalty to output the bounding box of the target in the search area, wherein the fusion proportion constant of each region proposal network is obtained by training; the corresponding inputs of the first, second and third region proposal network units are the shallow, medium and deep features of the target template and the search area.

[0042] Among them, the network structure of SiamRPN++, the ResNet50 network and the RPN network are known to those skilled in the art, and the embodiments of the present application do not repeat them.

[0043] Step 2: On the basis of the network structure of SiamRPN++, a sample generation module, a template linear update module, an online training module, a domain expansion module, a knowledge distillation module and a selection module are added;

[0044] Among them, taking the first and second frames as an example: the target template of the first frame and the second frame image are sent into SiamRPN++, and the normal tracking result and the confidence are obtained after calculation by the original algorithm.

[0045] Step 3: Enter the sample generation module:

[0046] The tracking result is sent into the sample generation library, and a certain proportion of positive and negative samples is generated according to the confidence, and the confidence is between 0 and 1.

[0047] Among them, assuming that the tracking result confidence is 0.8, a certain ratio of 1:2 of positive and negative samples is generated, a total of 9, the total number of samples is artificially set, and the number is generally not more than 20 to prevent subsequent online training from being too slow. The generation of positive and negative samples is to randomly move the tracking box, and the intersection over union of the tracking box and the tracked target is greater than a certain threshold, and the confidence threshold of the positive sample is 0.8; if it is lower than 0.1, it is set as a negative sample, and the positive and negative samples are temporarily stored.

[0048] Step 4: Enter the template linear update stage;

[0049] The new template adopts linear weighting of the target feature tracked in the last frame and the target template of the initial frame.

[0050] Take the first and second frames as examples, after step 3 ends, the high-confidence result tracked out by the second frame is sent to the backbone network to complete a feature extraction, and the extracted feature is linearly weighted with the target feature of the first frame to serve as the target template of the third frame. The linear weighting formula is as follows:

[0051] z = (1-Lr) x z0 + Lr x z

[0052] wherein z0 represents the target feature of the first frame, z represents the high-confidence target feature generated by the last frame, x represents multiplication, Lr represents an update rate, and is generally set as a constant, for example, 0.0102.

[0053] Step 5: entering an online training phase;

[0054] First, the RPN network model of the SiamRPN++ network is copied as a student network, and the original RPN network is frozen as a teacher network, the positive and negative sample library generated in step 3 is used as data with labeled information, and the student network is continuously trained using cross-entropy loss, and the training mode is improved according to the original SiamRPN++ training mode;

[0055] The training mode takes the first and second frames as examples, and specifically, the target of the first frame is used as the center to crop the template, and the size is adjusted to 127x127. Similarly, the second frame image is cropped on the current frame, and the size is twice that of the template, and then the size is adjusted to 255x255. The loss uses cross-entropy and regression loss (smooth L1 loss), the learning rate is 0.001, the period is one round, and the training process is optimized using the SGD optimizer.

[0056] The cross-entropy and regression loss is well known to those skilled in the art, and the embodiments of the present application do not repeat it.

[0057] Step 6: entering a domain expansion module;

[0058] The feature loss function is used to maintain the decision boundary of the classification branch of the student network on the original target feature. For example, Figure 3 Take the first and second frames as examples, after step 5, the positive and negative sample library of the second frame is extracted by the backbone network, and then passes through the deep related convolution part of the classification and regression branches of the teacher and student networks to obtain the features before the input Head.

[0059] The features before the input Head obtained by the two networks and the features extracted by the backbone network are calculated by mean square error, and the feature change loss is used to reduce the difference in feature mapping between the initial target template and the new target tracking result. The feature loss function is as follows:

[0060]

[0061] In the formula, x is the feature map of the image extracted by the backbone network; The weights of the teacher model and the student model, respectively; Indicates the feature before the classifier in the RPN network; F(·) is the mean square error (MSE); and a is a hyperparameter used to balance the loss.

[0062] Step 7: Enter the selection module;

[0063] According to the comparison of the classification responses tracked by the trained teacher and student networks in step 6, it is determined whether the knowledge distillation process is performed in the next training.

[0064] Among them, taking the second and third frames as examples, the student network trained by the positive and negative samples generated by the second frame is used to perform tracking calculation on the third frame image with the teacher network, and then the confidence of the tracking results of the two networks is compared. If the confidence obtained by the teacher network is greater than that of the student network, the teacher network guides the next training of the student network; otherwise, the teacher network does not guide the next training of the student network.

[0065] Step 8: Enter the knowledge distillation module;

[0066] In order to maintain the prediction ability of the student network for the past appearance information of the tracked target, a distillation loss is introduced as an additional supervision in the old model to prevent catastrophic forgetting, such as Figure 4 .

[0067] Among them, when a new video frame is sent into the twin tracking module for tracking, the student model parameters trained online in steps 5 and 6 are still used to compare the confidence of the tracking results of the teacher network and the student network, as in step 7. If the confidence of the teacher network is greater than that of the student network, the distillation loss training is performed; otherwise, the distillation loss training is not performed.

[0068] Taking the first, second and third frames as examples, the prediction results of the third frame by the teacher network and the student network are recorded as p T and p S , respectively:

[0069]

[0070] In the formula, F(·) is a loss function, which aims to reduce the difference between the two distributions, and β is a hyperparameter used to balance the loss. p s' is part of p s , which is the student's prediction of the last frame information of the tracked target. y' T and y' S are transformed versions of p T and p S , called "soft labels". The converter is a modified softmax function, where:

[0071]

[0072] In the formula, T is a smoothing parameter, referred to as temperature. The higher the temperature, the softer the label, i.e. the flatter the probability distribution.

[0073] The classification responses of the teacher network and the student network are weighted and sent to a selection module to complete final regression of the tracking frame and the like.

[0074] Therefore, the entire tracking process can be briefly described as follows: step 1 is initialized once, and the RPN network is copied as a student model only once in the initial frame, and steps 2 to 8 are sequentially executed in the subsequent loop to complete the continuous tracking process.

[0075] The above description is only used to illustrate the principles and effects of the present application, and is not intended to limit the present application. Those skilled in the art can modify the above examples without departing from the spirit and scope of the present application.

[0076] In summary, the present application discloses a kind of twin network target tracking method based on incremental learning, and incremental learning is applied to the updating process of target tracking network model. Firstly, the RPN (region proposal network) of tracking network SiamRPN++ is copied as a student model, and the high-confidence target generated in the tracking process is used as a small sample set for online training. Then, the small sample set generated by the last frame network is learned using the incremental learning method, and the student model is trained by domain expansion and knowledge distillation. Finally, the target information generated by the student network model and the target information generated by the teacher network model are dynamically weighted and fused to update the position. In view of the problems that the current model updating method lacks flexibility, training is complex and old knowledge is forgotten, the present application uses the incremental learning method to make the offline trained model have self-adaptive learning ability, not only effectively utilizes the historical information of the target in the tracking process, but also avoids large-scale offline training of the model, and improves the ability of twin network algorithm to handle target deformation and the like in the tracking process.

Claims

1. A twin network target tracking method based on incremental learning, characterized in that, The method comprises: On the basis of a twin network framework, taking the first frame target information as a template, and inputting the current frame into a tracking network to generate a new target position; generating a positive and negative sample library according to a high-confidence target sample generated in a tracking process, and fusing the target template in a linear updating manner; Taking the positive and negative sample library generated by the high-confidence target as new samples, copying the RPN part of the twin network framework into a student network to perform incremental training, the incremental training comprising domain expansion and knowledge distillation of the student network, and the purpose of the knowledge distillation being to prevent model forgetting of old knowledge through distillation loss; When a next frame image is input, the student network trained by the previous frame and the teacher network, i.e., the original network, are used to perform tracking again, the confidence of the target generated by the teacher and student networks is judged according to a classification response value, the classification feature maps of the student and teacher networks are linearly weighted and fused, and a high-confidence target sample is generated again; the sample library is further updated, the student network is incrementally trained, and continuous tracking is realized; In the initial frame, the RPN network of SiamRPN++ is copied as a student network, and the original RPN network is frozen as a teacher network for guiding the student network to learn; in this process, some layers of the classification branch of the student network are frozen to maintain its decision boundary; in addition, a feature change loss is used to reduce the difference in feature mapping between the target template and the new target tracking result. In the formula, x is a feature map of an image extracted by a backbone network; Respectively, the weights of the teacher model and the student model of the RPN part; Indicates the feature before the classifier in the RPN; F(·) is the mean square error (MSE); and a is a hyperparameter used to balance the loss. 2.The twin network target tracking method based on incremental learning according to claim 1, wherein, The twin network framework is SiamRPN++. 3.The twin network target tracking method based on incremental learning according to claim 1, characterized in that, The generation of the positive and negative sample library specifically comprises: A high-confidence target is taken as the center, and a tracking box is randomly moved, so that a positive sample is obtained when the intersection over union with the tracking target is greater than a certain threshold, and a negative sample is obtained when the intersection over union is less than a certain threshold; the positive and negative ratio and the sample quantity of the positive and negative sample library are dynamic values, which are determined by the tracking response size obtained by the algorithm, i.e., when a high-confidence tracking result is generated, the positive sample ratio and quantity are increased; when a low-confidence tracking result is generated, the negative sample ratio and quantity are increased; when a new result is generated in the next frame, the sample library is replaced. 4.The method of claim 1, wherein, The linear updating manner of the generation of the target template specifically comprises: z = (1-Lr) x z0 + Lr x z wherein z0 represents the target feature of the first frame, z represents the high-confidence target feature generated in the previous frame, x represents multiplication, and Lr represents an updating rate, which is set as a constant. 5.The method of claim 1, wherein, The knowledge distillation process comprises: The teacher network and the student network are simultaneously input with the same frame of image, and the prediction results are recorded as p T and p S In order to enable the classification branch of the student network to learn new target appearance feature information, the traditional cross-entropy loss is used to optimize In addition, in order to maintain the prediction ability of the student network to the past appearance information of the target, p T is used to guide the training process; in addition, the following distillation loss is introduced in the old model as additional supervision to prevent catastrophic forgetting: where F(·) is a loss function aiming to reduce the difference between the two distributions, β is a hyper-parameter used to balance the loss; p s' is part of p s , is the prediction of the previous frame of information of the student network on the target to track, y' T and y' S are transformed versions of p T and p S , called "soft labels", the converter is a modified softmax function, where: wherein T is a smoothing parameter, referred to as temperature; the higher the temperature, the softer the label, i.e., the flatter the probability distribution. 6.A twin network target tracking method based on incremental learning, characterized in that, The modules and functions comprise: A sample generation module, configured to input an initial frame template and a current frame image into a network to obtain a tracking result, select a high-confidence target as a positive sample, and generate a positive and negative sample library according to the intersection over union; A template linear updating module, configured to input an initial frame template and a high-confidence target feature obtained through subsequent tracking into a network in a linear updating manner, adjust the template information to update the target feature, and always give priority to the information of the initial template in this module; The training module includes a domain expansion module and a knowledge distillation module. The training module is used to copy the RPN network of SiamRPN++ as the student network in the initial frame. The original RPN network is frozen as the teacher network to guide the student network learning. The student network is then trained online. The loss function is cross entropy loss and regression loss. Feature loss function and distillation loss function are added to the domain expansion and knowledge distillation branches respectively. The target is tracked after training. The domain expansion module of the submodule is used to freeze certain layers of the classification branch of the student network during training to maintain its decision boundary. Feature change loss is used to reduce the difference in feature mapping between the initial target template and the new target tracking result. The knowledge distillation module of the submodule is used to maintain the original feature extraction capability and the student network's ability to predict the target's past appearance information. Distillation loss is introduced in the old model as additional supervision to prevent catastrophic forgetting. The selection module is used to complete the subsequent calculation operations of the twin network output tracking box and to fuse the classification branches of the student network and the teacher network. The fusion method here can be linear weighted.

Images