A boundary-aware target tracking model and target tracking method

Abstract

A boundary-aware target tracking model and method includes a backbone feature extraction network, a discriminant model based on a deformable Transformer, and a tracking head network. The first backbone feature extraction network takes a template set as input, extracts the backbone features of the template set, and outputs one path to the discriminant model and another path to the PrRoI Pooling component of the tracking head network. The second backbone feature extraction network takes a search set as input, extracts the backbone features of the search set, and inputs one path to the tracking head network and another path to the discriminant model. The discriminant model includes an encoder, a decoder, and a discriminant filter. The weights of the discriminant filter are trained and determined using an online learning method. The weights are used as convolution kernels and convolved with the output of the decoder to obtain a response map of the tracking target. The high positions in the response map represent potential tracking target positions. The present invention can track specific targets in the input video in real time, thereby obtaining accurate candidate frames.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a boundary-aware target tracking model of a deformable Transformer and a target tracking method based on the model. Background Art

[0002] Object tracking plays a vital role in the field of computer vision. Object tracking technology is widely used in video surveillance, intelligent transportation, human-computer interaction, drones, and other fields. Applying object tracking technology to video surveillance can better detect and track unusual events, effectively reducing or preventing the occurrence of abnormal behavior. This makes video surveillance more intelligent and effective, with profound implications for both social security and industrial production. Applying object tracking to drones allows them to autonomously track, locate, and process events without human intervention. For robots to interact with humans, they must first know where humans are and be able to track their locations, enabling more intelligent applications. These intelligent technologies have profound implications for public safety, national defense, aerospace, and other fields. Through years of tireless research by numerous researchers, many excellent object tracking algorithms have been developed. In recent years, with the advancement of video acquisition equipment and the increasing sophistication of video network transmission, object tracking technology in video has garnered increasing attention. Video object tracking has become a key research topic across major university laboratories, research institutes, and major companies worldwide.

[0003] Existing target tracking methods are primarily based on sliding window trackers. These methods primarily generate candidate points in a pre-generated feature map and then continuously slide a window across the feature map to find a suitable target box. This method has demonstrated superior performance and computational efficiency in the tracking field, and has therefore attracted widespread attention. However, sliding window trackers have the following issues:

[0004] First, due to the local receptive field of convolutional networks, using features corresponding to candidate points to determine candidate target locations is relatively limited. Sliding window-based trackers neglect effective exploration of the target's boundary content, resulting in inaccurate predicted target bounding boxes. Sliding window trackers first utilize a Siamese network to extract backbone features from the template and search image, then perform correlation operations to obtain a feature response map of the target. Based on each pixel position in the feature response map predicted by this method, the sliding window tracker generates a large number of candidate points and then employs classification and regression networks to predict the target's tracking bounding box. Recently popular trackers such as SiamRPN++ and SiamFC++ first utilize a classification network to predict candidate target locations and then employ a regression network to refine the predicted bounding box. However, these methods use the features of each grid point in the correlation response map as input during computation. These features often contain background information, which can affect model performance. Furthermore, these methods neglect effective exploration of the target's boundary information.

[0005] Secondly, sliding window trackers, due to their lack of an online weight update mechanism, perform slightly worse than trackers based on online learning. These trackers are trained offline end-to-end on large-scale image pairs. However, in object tracking, the shape of the target changes in every frame of a video sequence due to challenging factors such as viewpoint changes, illumination variations, and rapid motion. Traditional sliding window trackers, lacking an online learning mechanism, are prone to drifting to the location of interfering objects. Summary of the Invention

[0006] Because existing sliding window-based trackers ignore the extraction of target boundary features, the present invention provides a boundary-aware network that can pay more attention to the target's boundary features, allowing sliding window-based trackers to more accurately predict the target's bounding box. Furthermore, the present invention provides a deformable Transformer model that is used to enhance the template features and test image features of existing sliding window trackers. To improve the robustness of the entire tracker, an online learning mechanism is introduced into the classification branch to enhance the classification score of the sliding window tracker.

[0007] The present invention proposes a boundary-aware target tracking model, which includes a backbone feature extraction network, a discriminant model based on a deformable Transformer, and a tracking head network. There are two backbone feature extraction networks. The first backbone feature extraction network takes a template set as input, extracts the backbone features of the template set, and outputs one path to the discriminant model and the other path to the PrRoI of the tracking head network. Pooling component; the second backbone feature extraction network takes the search set as input, extracts the backbone features of the search set, and then inputs one path into the tracking head network and the other path into the discriminant model; the template set and the search set are sets of images for tracking the target; the discriminant model is trained using an online learning method, including an encoder, a decoder, and a discriminant filter, the encoder is used to enhance the backbone features of the template set and the feature response of the tracked target, the decoder takes the output of the encoder and the backbone features of the search set as input, outputs the enhanced features, and transmits the location information of the target, the output of the decoder is divided into two paths, one of which is input to the discriminant filter, which is trained using an online learning method to determine the weight of the discriminant filter, and the weight is used as a convolution kernel to convolve with the other output of the decoder to obtain a response map of the tracked target, where the high position of the response map represents the potential tracking target position; the tracking head network is trained using an offline learning method, including a classification branch and a regression branch, the classification branch is used to calculate boundary-aware classification predictions, and the regression branch is used to predict boundary-aware regression predictions; the online learning and offline learning frameworks are both formulated under a unified multi-task learning framework.

[0008] Furthermore, the backbone feature extraction network is implemented using the deep residual network ResNet-50, and the outputs of Conv-3 and Conv-4 of the first deep residual network ResNet-50 are used as the backbone features of the template set, and the outputs of Conv-3 and Conv-4 of the second deep residual network ResNet-50 are used as the backbone features of the search set.

[0009] Furthermore, in the discriminant model based on the deformable Transformer, the backbone features of the template set are passed through a convolutional network to generate query features and convolution features; then, the query features are passed through a linear mapping layer to generate the relative offset sum of the target, and the convolution features are combined with the sum through linear mapping to generate a deformable output of the target.

[0010] Furthermore, a task-specific interactive head network is introduced into the tracking head network to enhance the interaction between classification and regression tasks.

[0011] Furthermore, a BorderAlign component is introduced in the classification branch and the regression branch to adaptively distinguish the boundary information of the target from the search image.

[0012] On the other hand, the present invention proposes a boundary-aware target tracking method, which realizes accurate prediction of target candidate frames based on the above-mentioned boundary-aware target tracking model, including: constructing a training data set; based on the data set, training the discriminant model based on the deformable Transformer to obtain the online learning score of the target; based on the data set, training the classification branch of the target tracking model to obtain the classification score of the target; based on the data set, training the regression branch of the target tracking model to obtain the regression score of the target; fusing the online learning score calculated by the discriminant model and the classification score to obtain the final classification score; using the regression score to predict the regression offset of the target to obtain the final tracking frame.

[0013] Furthermore, the training of the discriminant model based on the deformable Transformer to obtain the online learning score of the target includes: using a zero-padding operation to make the online learning score map and the classification score map have the same dimension size, and fusing the learned online learning score with the classification score of the tracking head network.

[0014] Furthermore, the training of the discriminant model based on the deformable Transformer to obtain the online learning score of the target includes: inputting the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; inputting the backbone features into the convolutional network to obtain convolutional features; performing data enhancement on the convolutional features, including flipping, rotating, cropping, and scaling, to obtain enhanced convolutional features; inputting the data-enhanced convolutional features into the deformable Transformer model, and then into the discriminant filter to train the weights of the discriminant filter; obtaining the target position by selecting a discriminative learning loss and using an iterative optimization solution; convolving the weights of the trained filter as a convolution kernel with the search feature to obtain a score response map of a specific tracking target; and performing a zero-filling operation on the score response map to obtain the online learning score of the target.

[0015] Furthermore, the classification branch of the target tracking model is trained to obtain a classification score of the target, including: inputting the constructed training data into a backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; inputting the extracted backbone features into PrRoI Pooling to extract the pooled features of the target; inputting the extracted backbone features into a convolutional network to obtain convolutional features; obtaining relevant features by element-wise multiplying the pooled features of the target with the convolutional features; and inputting the extracted relevant features into a boundary perception network to obtain the classification score of the target.

[0016] Furthermore, the regression branch of the target tracking model is trained to obtain the regression score of the target, including: inputting the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; inputting the extracted backbone features into PrRoI Pooling to extract the pooled features of the target; inputting the extracted backbone features into the convolutional network to obtain the convolutional features; obtaining the relevant features by element-wise multiplying the pooled features of the target with the convolutional features; inputting the extracted relevant features into the boundary perception network to obtain the boundary perception regression score of the target; the regression branch incorporates BorderAlign to fine-tune the predicted target position, thereby obtaining an accurate target frame.

[0017] This paper proposes an improved tracking method for sliding window trackers. This method can track specific targets in input video in real time, thereby obtaining accurate candidate boxes. This model can fully train on class-imbalanced data and achieve relatively accurate target tracking without relying on techniques such as data augmentation. This method can be embedded in a camera or loaded onto a GPU server for network access. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram of the boundary-aware target tracking model of the present invention.

[0019] Figure 2 This is a block diagram of the deformable Transformer encoder model of the present invention.

[0020] Figure 3 It is a flow chart of the present invention for training the discriminant model.

[0021] Figure 4 It is a flow chart of the classification branch of the training tracker of the present invention.

[0022] Figure 5 It is a flow chart of the regression branch of the training tracker of the present invention. DETAILED DESCRIPTION

[0023] Attachment Figure 1 The figure shows the block diagram of the boundary-aware target tracking model of the present invention. In the proposed tracking framework, it mainly includes two parts: (1) a discriminant model framework based on Transformer, which is trained using online learning. (2) a tracking head network composed of a classification branch and a regression branch, which is trained using offline learning. In particular, the tracking head network introduces a task-specific interactive module to enhance the interaction between the classification task and the regression task. At the same time, a BorderAlign component is introduced in the classification branch and the regression branch to explore the boundary features of the target. At the same time, the proposed online learning and offline learning frameworks are both formulated under a unified multi-task learning framework.

[0024] like Figure 1 As shown in Figure 1, the template set and the search set are input to the backbone feature extraction network, where ResNet-50 is used as the feature extractor. The backbone feature extraction network structure and parameters used in the present invention are shown in Table 1.

[0025] Table 1 Backbone feature extraction network structure and parameters used in the present invention

[0026]

[0027] The deep residual network is composed of many residual modules, which solves the problem of deep model degradation. The resolution of the input image is relatively high, and the present invention uses a resize operation to crop it to a size of 352*352. The deep residual network is used as the backbone feature extraction network to extract features from the cropped image, and the extracted feature dimensions are 22*22 and 44*44. Here, the outputs of Conv-3 and Conv-4 of ResNet-50 are used as backbone features. The entire backbone feature extraction network uses target detection and tracking data sets as training data. During the test process, the learned backbone feature extraction network weights remain fixed. The extracted input image backbone features are input into the discriminant model and the tracking head network respectively.

[0028] In the discriminative model, the extracted backbone features are first fed into a convolutional network to reduce the number of backbone feature channels from 1024 to 512. These reduced-channel convolutional features are then fed into a deformable transformer to explore the rich temporal information of the target. Specifically, the convolutional features of the template set are fed into the Transformer encoder to enhance the target's feature response. The output of the Transformer encoder and the backbone features of the search set are then fed into the Transformer decoder to convey the target's location information, simplifying the target search process. The output features of the Transformer decoder are then fed into a discriminative filter to learn its weights. The learned discriminative filter weights are selected as convolution kernels and convolved with the decoder's output features to produce a target response map. Locations with high values in the response map represent potential target locations. To integrate the learned online learning scores with the classification scores of the tracking head network, zero padding is used to ensure that the online learning score map and the classification score map have the same dimensionality.

[0029] In the tracking head network, the extracted backbone features are first input into the PrRoI Pooling component to generate a modulation vector containing specific target content. The backbone features of the search set image in another branch are input into a convolutional network to generate convolutional features. The convolutional features are then convolved with the pooled features to generate the target's modulation vector. This modulation vector is the target's relevant feature, containing the tracking content of the specific target. These relevant features are then input into the classification branch and the regression branch: the classification branch calculates boundary-aware classification predictions, while the regression branch calculates boundary-aware regression predictions. A boundary alignment network is introduced for both classification and regression predictions. This introduced boundary alignment network includes a BorderAlign operation, which extracts target boundary features and adaptively distinguishes target boundary information from the search image. In boundary-aware classification prediction, the portion of the classification branch that does not include the boundary-aware module is used to predict a coarse classification score, while the portion of the classification branch that includes the boundary-aware module is used to predict a boundary-aware classification score. The online learning score calculated by the discriminant model and the classification score are then fused to generate the final prediction score. In the boundary-aware regression prediction, the regression branch does not include the boundary-aware network module to predict the coarse regression offset, while the regression branch includes the boundary-aware network module to predict the boundary-aware regression offset. Finally, the coarse regression prediction and the boundary-aware regression prediction are combined to obtain the final tracking box.

[0030] In order to optimize the proposed tracking model, the present invention uses multi-task loss to train on a large-scale training dataset. The classification and regression predictions calculated above are optimized using the following formula:

[0031]

[0032] in, represents the coarse classification loss, represents the rough regression loss, represents the boundary-aware classification loss, represents the boundary-aware regression loss. Here and are the corresponding coarse classification and regression predictions, and is the corresponding true value, and Represent the predicted rough regression score and the corresponding true value, respectively. and represent the boundary-aware regression score and the corresponding true value, respectively. and Represent the predicted centrality score and the corresponding true value, respectively. Denotes the discriminative learning loss. In actual training, The following training is used:

[0033]

[0034] Where is the hyperparameter, and The following training is used:

[0035]

[0036] Among them, is a hyperparameter, and in (1) The following training is used:

[0037]

[0038] Where a is a hyperparameter, and (1) The following training is used:

[0039]

[0040] in is a hyperparameter, and in (1) Cross entropy is used for training, which is defined as follows:

[0041]

[0042] In formula (1), Hinge loss is used to train the discriminative filter.

[0043] See also Figure 2 , which is the block diagram of the deformable Transformer encoder model of the present invention:

[0044] Main features of the template set Generate query feature z through convolutional network q And the convolution feature z c Then, query feature z q The relative offset Δp of the target is generated after the linear mapping layer and the Softmax layer qk and attention weight Δm qk , convolutional feature z c After linear mapping and Δp qk and Δm qk Combined to generate the cumulative sampling value d of the target. The calculation of the cumulative sampling value d in the Transformer encoder can be expressed as:

[0045]

[0046] Where k is the key sampling point; K is the total number of sampling points; W k is the convolution weight of the kth sampling point; p q is the convolution feature z c The reference point p in q The accumulated sampling value d is linearly mapped and then combined with the convolution feature z c The target deformable output T is accumulated. In general, the calculation of the entire deformable Transformer encoder can be summarized as:

[0047] T=Reshape(Lay.Norm(Linear(d)+z c )), (8)

[0048] Where T represents the output of the Transformer encoder; Lay.Norm represents layer normalization; Linear represents linear mapping; It is worth noting that in the entire Transformer encoder, the entire output feature T is consistent with the original input feature Have the same resolution size.

[0049] Figure 3 The flowchart of an embodiment of the present invention for training a discriminant model includes:

[0050] 1): Build a training dataset.

[0051] 2) Input the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image. This paper uses ResNet-50 as the backbone feature extraction network, and uses the outputs of its Conv_3 and Conv_4 as the input image features.

[0052] 3): Input the extracted input image features into the convolutional network to obtain convolution features.

[0053] 4) Perform data enhancement on the convolutional features, including flipping, rotating, cropping, and scaling, to obtain 10 sets of data-enhanced convolutional features.

[0054] 5): The data-enhanced convolutional features are input into the deformable Transformer model. The encoder in the Transformer model enhances the template features through the attention model, which is conducive to the learning of high-quality tracking models. At the same time, the decoder propagates tracking information from the previous template to the current frame, thereby simplifying the target search process.

[0055] 6) The Transformer-enhanced features are fed into the discriminative filter to train its weights. The discriminative filter is an end-to-end tracking architecture that fully leverages both target and background information for target prediction. By selecting a discriminative learning loss and employing an iterative optimization approach, the target position can be determined. The trained filter weights are used as convolution kernels and convolved with the search features to generate a score response map for the specific tracked target.

[0056] 7): Use zero padding operation on the score response map to obtain the target online learning score.

[0057] Figure 4 The flowchart of the classification branch of the target tracking model training of the present invention includes:

[0058] 1): Build a training dataset.

[0059] 2) Input the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image. This paper uses ResNet-50 as the backbone feature extraction network, and uses the outputs of its Conv_3 and Conv_4 as the backbone features.

[0060] 3) The extracted backbone features are fed into PrRoI Pooling to extract the pooled features of the target. PrRoIPooling does not involve any quantization or rounding, which solves the problem of the number of interpolated pixels N in RoI Align. Furthermore, the extracted backbone features are fed into a convolutional network to obtain convolutional features. Finally, the pooled features of the target are element-wise multiplied with the convolutional features to obtain the relevant features.

[0061] 4) The extracted relevant features are fed into the boundary-aware network to obtain the classification score of the target. The boundary-aware network introduces the BorderAlign operation to explicitly extract the features of the target boundary extreme points, which is efficient and does not introduce a lot of redundant background information.

[0062] Figure 5 The flowchart of the present invention for training the tracker regression branch includes:

[0063] 1): Build a training dataset.

[0064] 2) Input the constructed training data into the backbone feature extraction network to obtain input image features. This paper uses ResNet-50 as the backbone feature extraction network, and uses the outputs of its Conv_3 and Conv_4 as the backbone features.

[0065] 3) The extracted backbone features are fed into PrRoI Pooling to extract pooled features of the target. PrRoIPooling does not involve any quantization or rounding, which solves the problem of the number of interpolated pixels N in RoI Align. Furthermore, the extracted backbone features are fed into a convolutional network to obtain convolutional features. Finally, the pooled features are element-wise multiplied with the convolutional features to obtain the correlation features.

[0066] 4) The extracted relevant features are input into the boundary-aware network to obtain the boundary-aware regression score of the target. This regression branch is integrated with BorderAlign to fine-tune the predicted target position and obtain an accurate target bounding box.

Claims

1. A boundary-aware target tracking model, characterized in that It includes a backbone feature extraction network, a discriminant model based on a deformable Transformer, and a tracking head network; wherein, there are two backbone feature extraction networks, the first backbone feature extraction network takes the template set as input, extracts the backbone features of the template set, outputs one path to the discriminant model, and the other path outputs to the PrRoI of the tracking head network Pooling component; the second backbone feature extraction network takes the search set as input, extracts the backbone features of the search set, and then inputs one path into the tracking head network and the other path into the discriminant model; the template set and the search set are sets of images for tracking the target; the discriminant model is trained using an online learning method, including an encoder, a decoder, and a discriminant filter, the encoder is used to enhance the backbone features of the template set and the feature response of the tracked target, the decoder takes the output of the encoder and the backbone features of the search set as input, outputs the enhanced features, and transmits the location information of the target, the output of the decoder is divided into two paths, one of which is input to the discriminant filter, which is trained using an online learning method to determine the weight of the discriminant filter, and the weight is used as a convolution kernel to convolve with the other output of the decoder to obtain a response map of the tracked target, where the high position of the response map represents the potential tracking target position; the tracking head network is trained using an offline learning method, including a classification branch and a regression branch, the classification branch is used to calculate boundary-aware classification predictions, and the regression branch is used to predict boundary-aware regression predictions; the online learning and offline learning frameworks are both formulated under a unified multi-task learning framework.

2. The target tracking model according to claim 1, characterized in that The backbone feature extraction network is implemented using the deep residual network ResNet-50, and the outputs of Conv-3 and Conv-4 of the first deep residual network ResNet-50 are used as the backbone features of the template set, and the outputs of Conv-3 and Conv-4 of the second deep residual network ResNet-50 are used as the backbone features of the search set.

3. The target tracking model according to claim 1, wherein: In the discriminant model based on the deformable Transformer, the backbone features of the template set are converted into query features and convolution features through a convolutional network; then, the query features are converted into the relative offset Δp of the target through a linear mapping layer. qk and Δm qk , the convolution feature is linearly mapped with Δp qk and Δm qk Combined to generate a deformable output of the target.

4. The target tracking model according to claim 1, wherein: A task-specific interactive head network is introduced into the tracking head network to enhance the interaction between classification and regression tasks.

5. The target tracking model according to claim 1, wherein: A BorderAlign component is introduced in the classification branch and the regression branch to adaptively distinguish the boundary information of the target from the search image.

6. A boundary-aware target tracking method, the method being based on any boundary-aware target tracking model according to claims 1 to 5, achieving accurate prediction of target candidate frames, characterized in that: include: Build a training dataset; Based on the data set, training the discriminant model based on the deformable Transformer to obtain an online learning score of the target; Based on the data set, training the classification branch of the target tracking model to obtain a classification score of the target; Based on the data set, training the regression branch of the target tracking model to obtain a regression score of the target; Fusing the online learning score calculated by the discriminant model and the classification score to obtain a final classification score; The regression score is used to predict the regression offset of the target to obtain the final tracking frame.

7. The target tracking method according to claim 6, wherein: The training of the deformable Transformer-based discriminant model to obtain an online learning score of the target includes: using a zero-padding operation to make the online learning score map and the classification score map have the same dimension size, and fusing the learned online learning score with the classification score of the tracking head network.

8. The target tracking method according to claim 6, wherein: The training of the deformable Transformer-based discriminant model to obtain a target online learning score includes: Input the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; Inputting the backbone features into a convolutional network to obtain convolutional features; Performing data enhancement on the convolutional features, including flipping, rotating, cropping, and scaling, to obtain enhanced convolutional features; The data-augmented convolutional features are fed into a deformable Transformer model and then into a discriminative filter to train its weights. The target position is obtained by selecting a discriminative learning loss and using an iterative optimization solution. The trained filter weights are used as convolution kernels and convolved with the search features to obtain a score response map for a specific tracking target. The score response map is zero-filled to obtain the target online learning score.

9. The target tracking method according to claim 6, wherein: The training of the classification branch of the target tracking model to obtain a classification score of the target includes: Input the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; The extracted backbone features are input to PrRoI Pooling to extract the pooled features of the target; the extracted backbone features are input to the convolutional network to obtain the convolutional features; the relevant features are obtained by element-wise multiplication of the pooled features of the target and the convolutional features; The extracted relevant features are input into the boundary perception network to obtain the classification score of the target.

10. The target tracking method according to claim 6, wherein: The step of training the regression branch of the target tracking model to obtain a regression score of the target includes: Input the constructed training data into the backbone feature extraction network to obtain the backbone features of the input image; the backbone feature extraction network adopts ResNet-50, and the backbone features are the outputs of Conv_3 and Conv_4 of the ResNet-50; The extracted backbone features are input to PrRoI Pooling to extract the pooled features of the target; the extracted backbone features are input to the convolutional network to obtain the convolutional features; the relevant features are obtained by element-wise multiplication of the pooled features of the target and the convolutional features; The extracted relevant features are input into the boundary-aware network to obtain the boundary-aware regression score of the target; the regression branch is integrated into BorderAlign to fine-tune the predicted target position to obtain an accurate target frame.

Images