Robust Object Tracking Method and System Based on Multi-Stage Adaptive Network

Abstract

The present invention relates to a robust object tracking method and system based on a multi-stage adaptive network. The method comprises the following steps: S1, constructing an offline training data set; S2, training the offline training data set by using a multi-step gradient descent training method of meta-learning to obtain a meta-learning model; S3, constructing a multi-stage adaptive network, including three different adaptive sub-networks: a domain adaptive sub-network, a feature adaptive sub-network, and a location adaptive sub-network; S4, inputting the first frame with only the known target location in the test video into the domain adaptive sub-network to perform initial training on the constructed multi-stage adaptive network; S5, successively inputting the subsequent frames starting from the second frame in the test video into the feature adaptive sub-network, and using the multi-stage adaptive network to track the test video. This method and system are conducive to obtaining more stable, robust, and accurate tracking results.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a robust object tracking method and system based on a multi-stage adaptive network. Background Art

[0002] Object tracking is an indispensable branch in the field of computer vision and has wide applications in video surveillance, virtual reality, and traffic control. The task of object tracking is to continuously locate the object in consecutive frames only when the initial object position and size are given in the first frame. In recent years, object tracking based on deep learning has made significant progress due to its powerful feature representation ability. However, adaptively handling large appearance changes caused by factors such as illumination changes, scale changes, motion blur, deformation, fast motion, and background clutter remains a challenging problem.

[0003] A large amount of research work has utilized large-scale datasets to train tracking models to obtain general features. For example, TransT (X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, H. Lu, Transformer tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 8126–8135.) introduced the attention mechanism in transformers to replace the previous cross-correlation operation and established long-range relationships between features. MDNet (H. Nam and B. Han. Learning multi-domain convolutional neural networks for visual tracking. Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, pp. 4293-4302.) regarded object tracking as a binary classification task and used the positive and negative sample features drawn around the predicted object to update the model. To improve the robustness of the tracker, VITAL (Y. B. Song, C. Ma, X. H. Wu, L. J. Gong, L. C. Bao, W. M. Zuo, C. H. Shen, R. W. H. Lau, M. H. Yang. VITAL: Visual tracking via adversarial learning. Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, pp. 8990-8999.) introduced adversarial learning on this basis to enhance the positive sample features in the feature space to enrich the appearance information. However, the above tracking methods mainly rely on the features extracted by the source domain training model, which causes difficulties in learning generalizable and robust appearance features and adapting to new video sequences. Currently, some trackers introduce meta-learning to solve the domain adaptation problem.For example, Meta-tracker (E. Park and A. C. Berg. Meta-tracker: fast and robust online adaptation for visual object trackers. Proceedings of the European Conference on Computer Vision. 2018, pp. 587-604.) uses a gradient-based meta-learning method to provide a good initialization parameter for quickly adapting to the current task. However, these meta-learning-based trackers do not fully consider the robustness of features.

[0004] There has been a large amount of representative work laying the foundation for the field of object tracking. For example, SINT (R. Tao, E. Gavves, A. W. Smeulders, Siamese instance search for tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1420–1429.) calculates the similarity score between the initial target in the first frame and the candidate targets in the current frame through a learned matching function, and returns the highest response map as the predicted target. SiamFC (L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional siamese networks for object tracking. Proceeding of the European Conference on Computer Vision Workshop. 2016, pp. 850-865.) uses a fully convolutional siamese network to estimate the similarity between the template feature and the search region feature. In addition, there is a large amount of subsequent work. For example, SiamRPN (B. Li, J. Yan, W. Wu, Z. Zhu, X. Hu, High performance visual tracking with siamese region proposal network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8971–8980.) regards tracking as single-sample local detection, adds a region candidate network to the siamese network, improves the cross-correlation strategy, and obtains superior performance and more accurate object scale estimation. Ocean (Z. Zhang, H. Peng, J. Fu, B. Li, W. Hu, Ocean: Object-aware anchor-free tracking, in: Proceedings of the European Conference on Computer Vision, Vol. 12366, 2020, pp. 771–787.) introduces a feature alignment module to adapt to scale changes and learns an object-aware feature to correct inaccurate object positions, thus dynamically adapting to appearance changes.However, due to the lack of adaptability to handle appearance changes, the above-mentioned object tracking methods cannot be well generalized to the test tasks. Summary of the Invention

[0005] The object of the present invention is to provide a robust object tracking method and system based on a multi-stage adaptive network, which is conducive to obtaining more stable, robust and accurate tracking results.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a robust object tracking method based on a multi-stage adaptive network, including the following steps:

[0007] S1. Randomly extract two frames of pictures from the video sequence dataset. The first frame is used as the support set, and the second frame is used as the query set. Repeat this operation to obtain multiple pairs of support sets and query sets, and thus construct an offline training dataset;

[0008] S2. Use the multi-step gradient descent training method of meta-learning to train the offline training dataset to obtain a meta-learning model

[0009] S3. Construct a multi-stage adaptive network, including three different adaptive sub-networks: a domain adaptive sub-network, a feature adaptive sub-network, and a location adaptive sub-network. The domain adaptive sub-network uses meta-learning training to enhance the generalization ability of the tracking model for new video sequences. The feature adaptive sub-network uses a multi-layer multi-scale meta-classifier module and an adversarial attention mask module to capture robust and discriminative feature information to update the tracking model. The location adaptive sub-network refines the bounding box of the preliminary result to increase the tracking accuracy;

[0010] Spectrum 4. Initialize and train the constructed multi-stage adaptive network; input the first frame with only the known target position in the test video into the domain adaptive sub-network, and find the optimal parameter θ' suitable for the current video sequence through the stochastic gradient descent algorithm for the first frame and the meta-learning model obtained from offline training to update the meta-feature extractor, and then input the target feature vector of the first frame into the adversarial attention mask module, use the generated attention mask to capture the robust features in the original features, generate meta-adversarial features, and the generated loss is backpropagated to update the meta-feature extractor;

[0011] S5. Use a multi-stage adaptive network to track the test video; starting from the second frame, input the current frame of the test video into the multi-layer multi-scale meta-classifier module in the feature adaptive subnet to obtain the classification scores of multiple candidate target samples, and take the average as the target confidence score; if the confidence score is not greater than 0, it is considered that the tracking is unsuccessful, and the resampling operation is performed, that is, the search area range of sampling is expanded, the target position where the previous tracking was successful is used as the target center of the current frame, and positive and negative samples are re-collected; if the confidence score is greater than 0, the candidate sample with the largest score is used as the target position and positive and negative samples are collected; then, perform the bounding box regression operation, use the result after regression as the input of the position adaptive subnet, and obtain the final target tracking result through the feature fusion operation with the features of the first frame; during the tracking process, perform long-term update every set number of frames, input the feature vector of the tracking result into the adversarial attention mask module, and calculate the cross-entropy loss by obtaining the meta-adversarial features through adversarial discriminant training; while the short-term update is performed when the confidence score is not greater than 0, and it does not execute the adversarial attention discriminant module, but directly calculates the cross-entropy loss using the re-collected positive and negative samples.

[0012] Further, in step S2, calculate the loss at each step on the support set to obtain the parameters updated at each step, and then use the updated parameters to calculate the loss at each step on the corresponding query set, sum the losses on the query set for overall gradient descent, so as to find the initial parameter θ that the model can generalize. * 。

[0013] Further, in step S3, the domain adaptive subnet consists of a meta-learning model and a meta-feature extractor, with the input being the first-frame picture with only the known target position, and the output being the feature vector of the first-frame target; the feature adaptive subnet consists of a multi-layer multi-scale meta-classifier module and an adversarial attention mask module, where the multi-layer multi-scale meta-classifier module is composed of a feature prediction network, with the input being the subsequent frames starting from the second frame, and the output being the target confidence score of this frame; the adversarial attention mask module is composed of a meta-adversarial feature generator, with the input being the feature vector of the target tracking result, and the output being the meta-adversarial features; the position adaptive subnet consists of a bounding box regression and a feature fusion operation, with the input being the preliminary result after the bounding box regression, and the output being the accurate target tracking result after adaptation.

[0014] Furthermore, in the domain adaptation subnet, the tracking model extracts positive and negative sample features from the input first-frame image, calculates the confidence score through the softmax function, then calculates the loss using the cross-entropy loss function, and further obtains the gradient. The optimal parameters θ' of the current video sequence are found through the stochastic gradient descent algorithm and then used to update the meta-feature extractor. The extracted features are used for adversarial training in the adversarial attention mask module; the initialized model trained by meta-learning is used for domain adaptation to reduce network overfitting, and the convolutional layer parameters of the network are updated to quickly adapt to the current sequence; the generalization ability of the offline-trained meta-learning model is improved through domain adaptation, enabling the model to adaptively adjust the learning rate according to the given target and then find the optimal parameters; the optimal parameters of the current video sequence are obtained through the following formula:

[0015]

[0016] where D S is the training data from the first frame of the video sequence, represents the meta-learning model with parameters θ * and α is the learning rate, is the gradient value calculated for θ * according to the first-frame loss L; represents the prediction value obtained by passing the first-frame training data D S through the meta-learning model and calculating the corresponding loss using the cross-entropy loss, and then taking the derivative to obtain the corresponding gradient.

[0017] Furthermore, in the feature adaptation subnet, the input of the adversarial attention mask module is the feature vector of the target result. The attention mask generated by adversarial learning and the extracted features are used for Dropout operation to output the meta-adversarial features; among them, the adversarial attention mask module uses a high-order cost-sensitive loss function to calculate the loss for updating the network model parameters; the input of the multi-layer multi-scale meta-classifier module is the subsequent frame images starting from the second frame. The meta-feature extractor with weight sharing is used to extract features from the subsequent frame images, and a feature fusion operation is performed on the feature sizes output by different convolutional layers, and then the corresponding scores are output through the fully connected layer; the specific method is: construct a multi-layer multi-scale meta-classifier module to fuse features of different layers and different scales to adapt to complex tracking scenarios such as scale changes and deformations;

[0018] If the confidence score of the candidate samples output by the multi-layer multi-scale meta-classifier module is greater than 0, the candidate sample with the highest score is used as the input of the position adaptive subnet; if the confidence score is not greater than 0, short-term update is performed; during the tracking process, long-term update is performed every ten frames. Through the iterative training of the generator and the discriminator, an adversarial attention mask module is used to generate a mask based on adversarial learning, and the mask that maximizes the discriminator loss is selected as the latest mask; a position attention mechanism is introduced to obtain a more effective attention mask and capture robust features to maintain the change of the target appearance; during short-term update, a resampling operation is performed, that is, the search area is expanded and samples are re-acquired to avoid the situation where the target is not within the search area; the loss function adopted by the adversarial attention mask module is:

[0019] L(q,c) = -(c·(1 - q)·log(q)+(1 - c)·q·log(1 - q))

[0020] where c is the class label, taking values 0 or 1, and q is the predicted probability of the target object.

[0021] Furthermore, in the position adaptive subnet, the input is the preliminary result after bounding box regression, and then feature fusion is performed with the first frame label, and finally the refined accurate result is output; the specific method is: first, the confidence score obtained by the multi-layer multi-scale meta-classifier module is used to determine whether the prediction result is reliable; if the confidence score is greater than 0, the tracking is considered reliable, otherwise the tracking is considered unreliable; the position adaptive subnet uses bounding box regression to process the frames with reliable tracking. This network refines the bounding box through feature fusion of the regression result to handle scale changes and partial occlusions; the refined bounding box is used as the final tracking result, and then the obtained meta-adversarial features are used as the input in the adversarial mask module; the bounding box regression method is as follows:

[0022] f(z,v) = (G′ x ,G′ y ,G′ w ,G′ h )≈(G x ,G y ,G w ,G h )

[0023] where z is the parameter, v is the input feature vector, f is the mapping function, x, y, w, h are the bounding box coordinates, G represents the label, and G′ is the predicted value.

[0024] Furthermore, during offline training, the learning rate is 0.0001.

[0025] The present invention also provides a robust object tracking system based on a multi-stage adaptive network, including a memory, a processor, and computer program instructions stored on the memory and executable by the processor. When the processor runs the computer program instructions, the above method steps can be implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects: The method and system effectively utilize three effective sub-networks, namely, the domain adaptive subnet, the feature adaptive subnet, and the location adaptive subnet. The domain adaptive subnet and the feature adaptive subnet can obtain generalized and robust object features by combining the advantages of meta-learning and adversarial learning, while the location adaptive subnet can refine the object location and avoid the drift problem. In addition, the tracking method can effectively update and train the model online, reduce the error accumulation during the tracking process, and make the tracker more robust to challenging scenarios. Description of the Drawings

[0027] Figure 1 It is a flowchart of the implementation process of an embodiment of the present invention. Detailed Embodiments

[0028] The present invention will be further described below in conjunction with the drawings and embodiments.

[0029] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0030] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] As Figure 1 shown, this embodiment provides a robust object tracking method based on a multi-stage adaptive network, including the following steps:

[0032] S1. Randomly extract two frames of pictures from the video sequence dataset. The first frame is used as the support set, and the second frame is used as the query set. Repeat this operation to obtain multiple pairs of support sets and query sets, and thus construct an offline training dataset.

[0033] S2. Use the multi-step gradient descent training method of meta-learning to train the offline training dataset to obtain a meta-learning model Specifically, calculate the loss at each step on the support set to obtain the parameters updated at each step, and then calculate the loss at each step on the corresponding query set using the updated parameters. Sum the losses on the query set for overall gradient descent to find the initial parameter θ that the model can generalize * 。

[0034] S3. Construct a multi-stage adaptive network, including three different adaptive sub-networks: a domain adaptive sub-network, a feature adaptive sub-network, and a location adaptive sub-network. The domain adaptive sub-network uses meta-learning training to enhance the generalization ability of the tracking model for new video sequences. The feature adaptive sub-network uses a multi-layer multi-scale meta-classifier module and an adversarial attention mask module to capture robust and discriminative feature information to update the tracking model. The location adaptive sub-network refines the bounding box of the preliminary result to increase the tracking accuracy.

[0035] Among them, the domain adaptive sub-network consists of a meta-learning model and a meta-feature extractor. The input is the first frame image with only the known target location, and the output is the feature vector of the target in the first frame. The feature adaptive sub-network consists of a multi-layer multi-scale meta-classifier module and an adversarial attention mask module. The multi-layer multi-scale meta-classifier module is composed of a feature prediction network. The input is the subsequent frames starting from the second frame, and the output is the target confidence score of this frame. The adversarial attention mask module is composed of a meta-adversarial feature generator. The input is the feature vector of the target tracking result, and the output is the meta-adversarial feature. The location adaptive sub-network consists of a bounding box regression and a feature fusion operation. The input is the preliminary result after bounding box regression, and the output is the accurate target tracking result after adaptation.

[0036] S4. Conduct initial training on the constructed multi-stage adaptive network; input the first frame with only the known target location in the test video into the domain adaptive sub-network. Use the stochastic gradient descent algorithm to find the optimal parameter θ' suitable for the current video sequence for updating the meta-feature extractor with respect to the first frame and the meta-learning model obtained from offline training Then input the target feature vector of the first frame into the adversarial attention mask module, capture the robust features in the original features using the generated attention mask to generate meta-adversarial features, and the loss generated is backpropagated to update the meta-feature extractor.

[0037] S5. Use a multi-stage adaptive network to track the test video; input the current frame of the test video (starting from the second frame) into the multi-layer multi-scale meta-classifier module in the feature adaptive subnet to obtain the classification scores of multiple candidate target samples, and take the average as the target confidence score; if the confidence score is not greater than 0, it is considered that the tracking is unsuccessful, and the resampling operation is performed, that is, the search area range of sampling is expanded, the target position where the previous tracking was successful is used as the target center of the current frame, and positive and negative samples are re-collected; if the confidence score is greater than 0, the candidate sample with the largest score is used as the target position and positive and negative samples are collected; then, perform the bounding box regression operation, take the result after regression as the input of the position adaptive subnet, and obtain the final target tracking result through the feature fusion operation with the first frame; during the tracking process, in order to enable the model to have the adaptive ability to the target surrounding environment, perform a long-term update every ten frames, input the feature vector of the tracking result into the adversarial attention mask module, and calculate the cross-entropy loss through adversarial discriminant training to obtain the meta-adversarial feature; while the short-term update is performed when the confidence score is not greater than 0, which does not require the execution of the adversarial attention discriminant module, but directly calculates the cross-entropy loss using the re-collected positive and negative samples.

[0038] In the domain adaptive subnet, the tracking model extracts the positive and negative sample features from the input first-frame picture, calculates the confidence score through the softmax function, then calculates the loss using the cross-entropy loss function, and further obtains the gradient. The optimal parameters θ' of the current video sequence are found through the stochastic gradient descent algorithm and then used to update the meta-feature extractor. The extracted features are used for adversarial training in the adversarial attention mask module; use the initialized model trained by meta-learning for domain adaptation to reduce the overfitting of the network, and update the convolutional layer parameters of the network to quickly adapt to the current sequence; improve the generalization ability of the offline-trained meta-learning model through domain adaptation, enable the model to adaptively adjust the learning rate according to the given target, and then find the optimal parameters; the optimal parameters of the current video sequence are obtained through the following formula:

[0039]

[0040] where D S is the training data from the first frame of the video sequence, represents the meta-learning model with parameter θ * , α is the learning rate, is the gradient value calculated for θ * according to the first-frame loss L; represents the first-frame training data D S passes through the meta-learning model to obtain the predicted value and calculate the corresponding loss using the cross-entropy loss, and then take the derivative to obtain the corresponding gradient.

[0041] In the feature adaptive subnet, the input of the adversarial attention mask module is the feature vector of the target result. The attention mask generated by adversarial learning and the extracted features are used for Dropout operation to output the meta-adversarial features. Among them, the adversarial attention mask module uses a high-order cost-sensitive loss function to calculate the loss for updating the network model parameters. The input of the multi-layer multi-scale meta-classifier module is the subsequent frame images starting from the second frame. The meta-feature extractor with weight sharing is used to extract features from the subsequent frame images, and a feature fusion operation is performed on the feature sizes output by different convolutional layers. Then, the corresponding scores are output through the fully connected layer. The specific method is as follows: construct a multi-layer multi-scale meta-classifier module to fuse features of different layers and different scales to adapt to complex tracking scenarios such as scale changes and deformations. Specifically, when the current frame is input into this module, the feature sizes of different convolutional layers can be extracted. The first layer is 96*25*25, the second layer is 256*5*5, and the third layer is 512*3*3. The features after three-layer convolution are fused and input into the fully connected layer to obtain the target confidence score for judging the update method. If the confidence score of the candidate sample output by the multi-layer multi-scale meta-classifier module is greater than 0, the candidate sample with the highest score is used as the input of the position adaptive subnet. If the confidence score is not greater than 0, short-term update is performed. During the tracking process, long-term update is performed every ten frames. Through the iterative training of the generator and discriminator, the adversarial attention mask module generates a mask based on adversarial learning, and the mask that maximizes the discriminator loss is selected as the latest mask. The position attention mechanism is introduced to obtain a more effective attention mask and capture robust features to maintain the change of the target appearance. During short-term update, a resampling operation needs to be performed, that is, the search area is expanded and samples are re-collected to avoid the situation where the target is not within the search area. The loss function adopted by the adversarial attention mask module is:

[0042] L(q,c) = -(c·(1 - q)·log(q)+(1 - c)·q·log(1 - q))

[0043] where c is the class label (taking values 0 or 1), and q is the predicted probability of the target object.

[0044] In the position adaptive subnet, the input is the preliminary result after bounding box regression, and then it is feature fused with the first frame label, and finally the refined accurate result is output. The specific method is as follows: First, the confidence scores obtained by the multi-layer multi-scale meta-classifier module are used to determine whether the prediction result is reliable. If the confidence score is greater than 0, the tracking is considered reliable, otherwise the tracking is considered unreliable. The position adaptive subnet uses bounding box regression to process the frames with reliable tracking. This network refines the bounding box through feature fusion to handle scale changes and partial occlusions to a certain extent. The refined bounding box is used as the final tracking result, and then the obtained meta-adversarial features are used as the input in the adversarial mask module. In practical applications, the adversarial attention mask module can also be used once per frame, which can improve the tracking accuracy. The bounding box regression method is as follows:

[0045] f(z,v)=(G′ x ,G′ y ,G′ w ,G′ h )≈(G x ,G y ,G w ,G h )

[0046] where z is a parameter, v is an input feature vector, f is a mapping function, x, y, w, h are the bounding box coordinates, G represents the label, and G' is the predicted value.

[0047] In this embodiment, the learning rate is 0.0001 during offline training.

[0048] This embodiment also provides a robust object tracking system based on a multi-stage adaptive network, including a memory, a processor, and computer program instructions stored on the memory and executable by the processor. When the processor runs the computer program instructions, the above method steps can be implemented.

[0049] In this embodiment, the OTB100 dataset is used for comparative verification. Table 1 shows the comparison results between the method proposed in the present invention and other object tracking methods on the OTB100 dataset. It can be seen from Table 1 that the method of the present invention has higher accuracy and robustness compared with other object tracking methods, specifically manifested as the best precision and success rate.

[0050] Table 1

[0051]

[0052]

[0053] In Table 1, TransT corresponds to the method proposed by X. Chen et al. (X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, H. Lu, Transformer tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8126–8135.)

[0054] SiamRCNN corresponds to the method proposed by P. Voigtlaender et al. (P. Voigtlaender, J. Luiten, P. H. S. Torr, B. Leibe, Siam R-CNN: visual tracking by re-detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 6577–6587.)

[0055] DaSiamRPN corresponds to the method proposed by Z. Zhu et al. (Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, W. Hu, Distractor-aware siamese networks for visual object tracking, in: Proceedings of the European Conference on Computer Vision, Vol. 11213, 2018, pp. 103–119.)

[0056] SiamRPN corresponds to the method proposed by B. Li et al. (B. Li, J. Yan, W. Wu, Z. Zhu, X. Hu, High performance visual tracking with siamese region proposal network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8971–8980.)

[0057] GradNet corresponds to the method proposed by P. Li et al. (P. Li, B. Chen, W. Ouyang, D. Wang, X. Yang, H. Lu, Gradnet: Gradient-guided network for visual object tracking, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6162–6171.)

[0058] DeepSRDCF corresponds to the method proposed by M. Danelljan et al. (M. Danelljan, G. Hager, F. Shahbaz Khan, M. Felsberg, Convolutional features for correlation filter based visual tracking, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2015, pp. 58–66.)

[0059] SiamDW corresponds to the method proposed by Z. Zhang et al. (Z. Zhang, H. Peng, Deeper and wider siamese networks for real-time visual tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4591–4600.)

[0060] SiamFC corresponds to the method proposed by L. Bertinetto et al. (L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, P. H. Torr, Fully-convolutional siamese networks for object tracking, in: Proceedings of the European Conference on Computer Vision Workshops, 2016, pp. 850–865.)

[0061] UDT corresponds to the method proposed by N. Wang et al. (N. Wang, Y. Song, C. Ma, W. Zhou, W. Liu, H. Li, Unsupervised deep tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1308–1317.)

[0062] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one or more of the flows Figure 1 or a combination of multiple flows and / or blocks

[0064] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one or more of the flows Figure 1 or a combination of multiple flows and / or blocks

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide for implementing the specified functions in Figure 1 one or more of the flows Figure 1Steps of the functions specified in one or more boxes.

[0066] As mentioned above, it is only the preferred embodiment of the present invention, and it is not a limitation to the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A robust target tracking method based on a multi-stage adaptive network, characterized in that, It includes the following steps: S1. Randomly extract two frames of pictures from the video sequence dataset. The first frame is used as the support set, and the second frame is used as the query set. Repeat this operation to obtain multiple pairs of support sets and query sets, and thus construct an offline training dataset; S2. Use the multi-step gradient descent training method with meta-learning to train the offline training dataset to obtain a meta-learning model ; S3. Construct a multi-stage adaptive network, including three different adaptive sub-networks: a domain adaptive sub-network, a feature adaptive sub-network, and a location adaptive sub-network. The domain adaptive sub-network uses meta-learning training to enhance the generalization ability of the tracking model for new video sequences. The feature adaptive sub-network uses a multi-layer multi-scale meta-classifier module and an adversarial attention mask module to capture robust and discriminative feature information to update the tracking model. The location adaptive sub-network refines the bounding box of the preliminary result to increase the tracking accuracy; S4. Initialize and train the constructed multi-stage adaptive network; Input the first frame with only known target positions in the test video into the domain adaptation subnet, and use the stochastic gradient descent algorithm to optimize the first frame and the meta-learning model obtained through offline training Find the optimal parameter θ’ suitable for the current video sequence to update the meta-feature extractor. Then, input the target feature vector of the first frame into the adversarial attention mask module, use the generated attention mask to capture the robust features in the original features, generate meta-adversarial features, and the generated loss is backpropagated to update the meta-feature extractor; S5. Use the multi-stage adaptive network to track the test video; starting from the second frame, input the current frame of the test video into the multi-layer multi-scale meta-classifier module in the feature adaptive sub-network to obtain the classification scores of multiple candidate target samples, and take the average as the target confidence score; If the confidence score is not greater than 0, it is considered that the tracking is unsuccessful, and the resampling operation is performed, that is, the search area range of the sampling is expanded, the target position where the previous tracking was successful is used as the target center of the current frame, and positive and negative samples are re-collected; If the confidence score is greater than 0, the candidate sample with the largest score is used as the target position and positive and negative samples are collected; Then, perform the bounding box regression operation, use the result after regression as the input of the location adaptive sub-network, and obtain the final target tracking result through the feature fusion operation with the first frame; during the tracking process, perform long-term update every set number of frames, input the feature vector of the tracking result into the adversarial attention mask module, and obtain the meta-adversarial feature through adversarial discriminative training to calculate the cross-entropy loss; while the short-term update is performed when the confidence score is not greater than 0, and it does not execute the adversarial attention discriminative module, but directly calculates the cross-entropy loss using the re-collected positive and negative samples; In the step S2, calculate the loss at each step on the support set to obtain the parameters updated at each step, and then calculate the loss at each step on the corresponding query set using the updated parameters. Sum the losses on the query set for overall gradient descent, so as to find the initial parameter θ that the model can generalize * ; In the step S3, the domain adaptation subnet is composed of a meta-learning model and a meta-feature extractor. The input is the first frame image with only the known target position, and the output is the feature vector of the first frame target. The feature adaptation subnet is composed of a multi-layer multi-scale meta-classifier module and an adversarial attention mask module. The multi-layer multi-scale meta-classifier module is composed of a feature prediction network. The input is the subsequent frames starting from the second frame, and the output is the target confidence score of this frame. The adversarial attention mask module is composed of a meta-adversarial feature generator. The input is the feature vector of the target tracking result, and the output is the meta-adversarial feature. The position adaptation subnet is composed of bounding box regression and feature fusion operations. The input is the preliminary result after bounding box regression, and the output is the accurate target tracking result after adaptation; In the domain adaptive sub-network, the tracking model extracts positive and negative sample features according to the input first-frame picture, calculates the confidence score through the softmax function, then calculates the loss using the cross-entropy loss function, and further obtains the gradient. The optimal parameters θ' of the current video sequence are found through the stochastic gradient descent algorithm, and then used to update the meta-feature extractor. The extracted features are used for adversarial training in the adversarial attention mask module; use the initialized model obtained by meta-learning training for domain adaptation to reduce the overfitting of the network, and update the convolutional layer parameters of the network to quickly adapt to the current sequence; improve the generalization ability of the offline training meta-learning model through domain adaptation, enable the model to adaptively adjust the learning rate according to the given target, and then find the optimal parameters; the optimal parameters of the current video sequence are obtained through the following formula: Among them, D S is the training data from the first frame of the video sequence, represents the meta-learning model with parameter θ * , α is the learning rate, is the gradient value calculated for θ * according to the loss L of the first frame; represents that the first-frame training data D S passes through the meta-learning model to obtain the predicted value, calculate the corresponding loss using the cross-entropy loss, and then take the derivative to obtain the corresponding gradient; In the feature adaptive subnet, the input of the adversarial attention mask module is the feature vector of the target result. The attention mask generated by adversarial learning and the extracted features are used for Dropout operation to output meta-adversarial features. Among them, the adversarial attention mask module uses a high-order cost-sensitive loss function to calculate the loss for updating the network model parameters. The input of the multi-layer multi-scale meta-classifier module is the subsequent frame images starting from the second frame. The meta-feature extractor with weight sharing is used to extract features from the subsequent frame images, and a feature fusion operation is performed on the feature sizes output by different convolutional layers, and then the corresponding scores are output through the fully connected layer. The specific method is as follows: construct a multi-layer multi-scale meta-classifier module to fuse features of different layers and different scales to adapt to complex tracking scenarios such as scale changes and deformations. If the confidence score of the candidate sample output by the multi-layer multi-scale meta-classifier module is greater than 0, the candidate sample with the highest score is used as the input of the position adaptive subnet. If the confidence score is not greater than 0, short-term update is performed. During the tracking process, long-term update is performed every ten frames. Through the iterative training of the generator and the discriminator, the adversarial attention mask module generates a mask based on adversarial learning, and selects the mask that maximizes the discriminator loss as the latest mask. A position attention mechanism is introduced to obtain a more effective attention mask and capture robust features to maintain the changes in the target appearance. During short-term update, a resampling operation is performed, that is, the search area is expanded and samples are re-acquired to avoid the situation where the target is not within the search area. The loss function adopted by the adversarial attention mask module is as follows: L(q,c) = -(c·(1 - q)·log(q)+(1 - c)·q·log(1 - q)) where c is the class label, taking values 0 or 1, and q is the predicted probability of the target object; In the position adaptive subnet, the input is the preliminary result after bounding box regression, and then feature fusion is performed with the first frame label, and finally the refined accurate result is output. The specific method is as follows: first, the confidence score obtained by the multi-layer multi-scale meta-classifier module is used to determine whether the prediction result is reliable. If the confidence score is greater than 0, the tracking is considered reliable, otherwise the tracking is considered unreliable. The position adaptive subnet uses bounding box regression to process the frames with reliable tracking. This network refines the bounding box through feature fusion of the regression result to handle scale changes and partial occlusions. The refined bounding box is used as the final tracking result, and then the obtained meta-adversarial features are used as the input in the adversarial mask module. The bounding box regression method is as follows: f(z,v) = (G x ′, G y ′, G′ w , G h ′) ≈ (G x , G y , G w , G h ) where z is a parameter, v is the input feature vector, f is the mapping function, x, y, w, h are the bounding box coordinates, G represents the label, and G' is the predicted value.

2. The robust object tracking method based on a multi-stage adaptive network according to claim 1, wherein, The learning rate is 0.0001 during offline training.

3. A robust object tracking system based on a multi-stage adaptive network, characterized in that, It includes a memory, a processor, and computer program instructions stored on the memory and capable of being run by the processor. When the processor runs the computer program instructions, the method steps described in claim 1 or 2 can be implemented.

Images