An instance-aware backbone network training method
By introducing instance-aware dispersed constraints and inter-class variance constraints in the backbone network, the problem of insufficient intra-class feature extraction in visual tracking is solved, the tracking accuracy and success rate are improved, and the effective distinction between similar goals is achieved.
Patent Information
- Application Number
- CN202111670920.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-31
AI Technical Summary
It is difficult for existing backbone networks to extract in-class features of the same category instance in visual tracking, causing the tracker to drift in scenarios with a large number of similar instances of the same type, affecting the tracking accuracy.
By adding instance-aware dispersed constraints in the target classification backbone network, combining inter-class variance constraints, the network guides the network to increase inter-class and intra-class spacing at the same time, improves feature extraction capabilities, and obtains the optimal backbone network by using iterative forward propagation and backpropagation training. .
It improves the success rate and accuracy of visual tracking, enhances the ability to distinguish similar goals, and does not change the original network structure and training data, making it easy to use.
Smart Images

Figure CN114399661B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of multi-target tracking, and in particular relates to an instance-aware backbone network training method. Background Art
[0002] Visual tracking is a fundamental task in computer vision (CV) with applications such as video surveillance, robotics, and human-computer interaction. Its goal is to predict the position and shape of a target in subsequent video frames based on the target information specified in the first frame. In the past decade or so, visual tracking based on convolutional neural networks (CNNs) has made remarkable progress. However, designing a high-precision visual tracker remains a challenging task due to many factors, one of the most important of which is interference from similar targets (IST). In scenes with a large number of similar instances of the same class, IST often causes the tracker to drift, bringing great difficulties to visual tracking.
[0003] The mainstream structure of modern visual trackers can be divided into a feature extraction backbone network, a feature fusion module, and a bounding box prediction module. Traditional backbone networks are usually pre-trained under the image classification framework. This training method encourages the backbone network to only extract inter-class features of instances of different categories. However, visual tracking not only needs to distinguish between targets and background clutter belonging to other categories, but also needs to distinguish between instances belonging to the same category of targets. This requires the feature network to extract features that are both inter-class and intra-class instance distinguishing. Summary of the invention
[0004] The purpose of the present invention is to provide an instance-aware backbone network training method, which adopts a mainstream target classification backbone network and adds instance-aware dispersion constraints when training the network without changing the network structure, training data, and hyperparameters of the original backbone network, so as to solve the defect that the existing backbone network cannot extract intra-class features of instances of the same category for visual tracking.
[0005] The technical solution provided by the present invention to solve the technical problem is as follows:
[0006] The present invention provides an instance-aware backbone network training method, which is applied to multi-target tracking, including network model pre-training, specifically: inputting a multi-target tracking data set into a target classification backbone network to perform forward propagation to generate intermediate parameters of the network model; using the intermediate parameters to perform back propagation to update the intermediate parameters; when performing back propagation, by adding instance-aware dispersion constraints and combining the original inter-class variance constraints of the target classification backbone network to guide the target classification backbone network to increase the inter-class spacing and the intra-class spacing at the same time, thereby improving the success rate and accuracy of visual tracking; iteratively performing forward propagation and back propagation to obtain the optimal target classification backbone network.
[0007] The beneficial effects of the present invention are as follows: the present invention increases instance-aware dispersion constraints on the basis of the inter-class distinction constraints of the original backbone network, thereby guiding the target classification backbone network to increase both the inter-class spacing and the intra-class spacing, thereby improving the success rate and accuracy of visual tracking; at the same time, the backbone network used does not change the original network structure, initialization method, training data and hyperparameters, does not increase the workload of using visual tracking, and the method of use is extremely convenient. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The present invention will be further described below in conjunction with the accompanying drawings.
[0009] Figure 1 It is a flow chart of the network model pre-training method.
[0010] Figure 2 This is a schematic diagram of the feature distribution of AlexNet's conv5 and fc8 layers from left to right.
[0011] Figure 3 This is a schematic diagram of the feature distribution of the conv4_4, conv5_4, and fc8 layers of VGG19 from left to right.
[0012] Figure 4 This is a schematic diagram of the feature distribution of ResNet50's conv3_3, conv4_6, conv5_3, and fc8 layers from left to right. DETAILED DESCRIPTION
[0013] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0014] The backbone network used in modern trackers was originally used for image classification tasks, not visual tracking. Therefore, the features extracted using this type of backbone network are not fully applicable to visual tracking. Therefore, the prior art proposes some feature screening, fusion and adjustment methods to extract features that are beneficial to visual tracking. It has been proven that higher backbone network layers encode more semantic features, while lower backbone network layers extract more discriminative information, which can better separate the target from distractors with similar appearance. Therefore, the present invention uses features of both high-level and low-level networks during the tracking process. It is further found that only a portion of the backbone network neurons are relevant to the target tracking task. Based on this finding, a feature map screening method is proposed to remove noise and irrelevant feature maps; this method can reduce computational redundancy and improve tracking accuracy. In addition, the transformer-based method performs feature fusion by using an attention mechanism, which performs contextual feature enhancement based on self-attention and mutual attention. The attention mechanism helps to fuse the backbone network features to obtain useful semantic information for visual tracking. In addition, fine-tuning the backbone network using tracking data can partially improve the intra-class discrimination ability of the features. However, due to the relatively small amount of tracking data, this improvement is very limited, especially for methods such as DCF that require online training. The above methods mainly focus on modifying the features extracted by the classification backbone network to facilitate visual tracking, although they can partially compensate for the lack of features; however, this compensation is very limited. For example, through feature screening, fusion, adjustment and other strategies, it is difficult to enhance the intra-class discrimination ability that the original features lack but is very important for tracking tasks. The present invention aims to build a backbone feature network with instance-aware function.
[0015] Please refer to Figure 1 The present invention provides an instance-aware backbone network training method, which is applied to multi-target video tracking, including network model pre-training, specifically: inputting a multi-target tracking data set into a target classification backbone network to perform forward propagation to generate intermediate parameters of the network model; using the intermediate parameters to perform back propagation to update the intermediate parameters; when performing back propagation, by adding instance-aware dispersion constraints and combining the original inter-class variance constraints of the target classification backbone network to guide the target classification backbone network to increase the inter-class spacing and intra-class spacing at the same time; iteratively performing forward propagation and back propagation to obtain the optimal target classification backbone network.
[0016] Specifically, the multi-target tracking dataset uses a commonly used dataset for target tracking, and the target classification backbone network uses the mainstream original backbone network. Forward propagation refers to the order from the input layer to the output layer of the neural network, and the intermediate variable parameters of the model are calculated and stored in sequence. Backward propagation is to calculate and store the gradient of the intermediate variables and parameters of each layer of the neural network in the order from the output layer to the input layer according to the chain rule. Image classification networks usually consist of a series of convolutional layers and several fully connected layers (FC layers). These convolutional layers are often used as backbone feature networks for various visual tasks. The general operation of the backbone network is: an image is input to the backbone network and then passes through a series of convolutional layers. Each convolutional layer outputs a set of feature maps, which are also the input of the next convolutional layer. The features output from the last convolutional layer of the backbone network are merged or flattened into a feature vector. The feature vector outputs a category encoding vector through the last few FC layers, and we expect the classification network to output a correct one-hot classification vector. That is, the element corresponding to the input image category is 1, and the other elements are 0. This means that we require that all images belonging to the same category output the same classification vector after passing through the classification network. This requirement is passed to each convolutional layer of the backbone network through backpropagation during network training to limit each convolutional layer to output features as close as possible in the feature space for images belonging to the same category. In actual video tracking, the features of each network layer are densely distributed in the feature space. Moreover, the closer to the output layer, the more concentrated the feature distribution. The close distance between the features of the same and similar instances is not conducive to the visual tracking task. This is because unlike the image classification task, which only needs to distinguish between objects of different categories, the visual tracking task not only needs to distinguish between objects of different categories, but also needs to distinguish similar instances. If the features of these instances are too close in the feature space, it will bring difficulties to instance distinction. Visual tracking requires the backbone network to provide features that are easy to distinguish between objects of the same and different categories. We hope to retrain the backbone network with the same network structure on the same classification dataset (such as ImageNet) as the original backbone network. The features extracted by the backbone network can effectively distinguish between objects of different categories and different instances of the same class. To achieve this goal, we need to modify the training process of the classification network. By constructing an instance-aware dispersion constraint, the classification network is guided to output classification vectors with both intra-class and inter-class discriminative power, and the backbone network is made to learn how to extract features with both intra-class and inter-class discriminative power through back propagation. Through back propagation, the backbone network is also guided to generate features with similar characteristics. In actual tracking, as the training progresses, the features of similar targets gradually gather and the boundaries between categories gradually form. This shows that the features extracted by the backbone network have good inter-class discriminative power.
[0017] It should be noted that, in the actual network training iteration process, the number of iterations can be preset, and the preset value is set to 20 to 200 times according to the prediction results, in order to obtain a satisfactory target classification backbone network.
[0018] Furthermore, the instance-aware dispersion constraint is an intra-class variance constraint, which is used to guide the target classification backbone network to output classification vectors with larger variance for targets of the same type, so as to increase the intra-class distance of targets of the same type. The intra-class variance constraint, intra-class distance and regularization term jointly construct the intra-class loss function.
[0019] Specifically, in network training, back propagation is performed simultaneously using intra-class and inter-class loss functions. The intra-class loss function consists of intra-class variance, inter-class distance, and regularization. The intra-class variance constrains features belonging to the same category to be close, but the variance constraint increases the intra-class distance to facilitate correct tracking. The regularization term is used to constrain the feature center to be close to the origin to ensure that it can be activated.
[0020] For further information, please refer to Figure 2 , Figure 3 , Figure 4, the selected multi-target tracking dataset can be OTB-2015, LaSOT or UAV123, and the selected original target classification backbone network is AlexNet, VGG19 or ResNet50 and its optimized network. Specifically, OTB2015 analysis is currently the most widely used and authoritative database, containing 100 video sequences. LaSOT contains 1400 videos, with an average of 2512 frames per sequence, each frame is carefully inspected and manually labeled, and the results are visually inspected and corrected when necessary. In this way, approximately 3.52 million high-quality bounding box annotations can be generated. In addition, LaSOT contains 70 categories, each of which contains 20 sequences. LaSOT is the largest object tracking dataset with high-quality manual dense annotations to date. Unlike previous datasets, LaSOT provides visual bounding box annotations and rich natural language specifications, which have been proven to be beneficial for various visual tasks, including visual tracking, by encouraging and promoting the exploration of integrated visual and language features to achieve strong tracking performance. For the target classification backbone network, UAV123 is mainly composed of 91 drone videos, among which several video sequences are long and split into three or four shorter clips and used multiple times, so there are 123 groundtruths, including 20 long videos, namely UAV20L. AlexNet consists of 5 convolutional layers (conv) and 3 fully connected layers (fc), and the activation function uses ReLU. The entire network has more than 62 million trainable parameters. Although the classic AlexNet is very amazing in terms of recognition effect, these are all based on a large number of adjustments to hyperparameters. It does not propose a clear model design rule to guide the subsequent design of new network models, which also limits its migration ability. VGG is used as the basic module of the convolutional neural network model for feature extraction. One of the advantages of VGG over AlexNet: VGG proposes the idea of replacing network layers with basic blocks, which allows it to reuse these basic blocks when building deep network models. Unlike AlexNet, which refers to a single model, VGG includes multiple different models, including VGG-11, VGG-13, VGG-16, and VGG-19. The numbers 11, 13, 16, and 19 behind them are the number of network layers. VGG19 includes 19 hidden layers (16 convolutional layers and 3 fully connected layers), with a simple structure. It uses very small convolution kernels (3×33×3) to build convolutional neural network structures of various depths, which can achieve good recognition accuracy. Compared with VGG19, ResNet50 does not use a fully connected layer, but a global average pooling layer, which can reduce a large number of parameters.
[0021] Furthermore, the instance-aware backbone network training method uses the t-SNE algorithm for dimensionality reduction.
[0022] Specifically, t-SNE is a machine learning algorithm for dimensionality reduction. It is a nonlinear dimensionality reduction algorithm and is very suitable for reducing high-dimensional data to 2 or 3 dimensions for visualization.
[0023] Furthermore, the instance-aware backbone network training method also includes model fine-tuning and retraining, specifically: after target calibration on the target tracking video of the actual scene, the target classification backbone network that has been pre-trained by the network model is trained again, so as to tune the model parameters so that the target classification backbone network can adapt to the target tracking of the actual scene.
[0024] Specifically, models trained using common data sets may not be suitable for actual application scenarios. It is necessary to use some video tracking data from actual application scenarios to conduct in-depth training and parameter fine-tuning on the target in order to improve the pertinence and robustness of the network model. The process of retraining the network through model fine-tuning is as follows: first, the first frame image is taken as input, and the initial coordinates of the target are given. Then, the candidate window is selected for the target, and the features of the candidate window are used as input after convolution pooling. When the number of tracking video frames does not reach the pre-set long-time threshold and the evaluation score of the current target is less than zero: positive and negative samples are sampled for the target in the first frame image. , train the inclusion domain model, and train the parameters of the fully connected layer and the classifier; in the next frame image, extract the positive sample candidate window of the target, calculate the highest score among the candidate positive samples, if the highest score is greater than the set threshold, add this sample to the positive sample library, and use the inclusion domain model to determine the final target position; if the highest score is less than the set threshold, it means that the network parameters no longer meet the current tracking, so the positive and negative sample libraries are used to train and update the fully connected layer and classifier parameters again; when the number of tracking video frames reaches the pre-set long-term threshold, the positive and negative sample library data is used to train and update the parameters of the fully connected layer and the classifier.
[0025] The technical solutions provided by the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the methods and core ideas of the present invention. For those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. This specification should not be understood as a limitation on the present invention.
Claims
1. An instance-aware backbone network training method for multi-target video tracking, characterized in that: Including network model pre-training, specifically: The multi-target tracking dataset is input into the target classification backbone network to perform forward propagation to generate the intermediate parameters of the network model; the target classification backbone network is transmitted to each convolutional layer of the backbone network through back propagation during the network training process to limit each convolutional layer to output features as close as possible in the feature space for images belonging to the same category, thereby completing the inter-class target classification task of distinguishing targets of different categories; An instance-aware dispersion constraint is constructed to guide the classification network to output classification vectors with both intra-class and inter-class discriminability. Use the intermediate parameters to perform back propagation to update the intermediate parameters; when performing back propagation, the constructed instance perception dispersion constraint is combined with the original inter-class variance constraint of the target classification backbone network to guide the target classification backbone network to increase the inter-class spacing and intra-class spacing at the same time; Iteratively perform forward propagation and back propagation to obtain the optimal target classification backbone network; The instance-aware dispersion constraint is an intra-class variance constraint, which is used to guide the target classification backbone network to output a classification vector with a larger variance for targets of the same type, so as to increase the intra-class distance of targets of the same type; The intra-class variance constraint, the intra-class distance and the regularization term jointly construct an intra-class loss function; The target classification backbone network is AlexNet, VGG19 or ResNet50; The instance-aware backbone network training method adopts t-SNE algorithm for dimensionality reduction.
2. The instance-aware backbone network training method according to claim 1, characterized in that: The multi-target tracking dataset is OTB-2015, LaSOT or UAV123.
3. The instance-aware backbone network training method according to claim 1, characterized in that: The instance perception backbone network training method also includes model fine-tuning and retraining, specifically: after target calibration on the target tracking video of the actual scene, the target classification backbone network that has been pre-trained by the network model is trained again, so as to tune the model parameters so that the target classification backbone network can adapt to the target tracking of the actual scene.
