An end-to-end single target tracking method based on pyramid pooling attention mechanism
By introducing a pyramid pooling attention mechanism in a single-target tracker, the problem of poor performance in the existing technology when dealing with target scale changes and deformation is solved, and more refined feature extraction and information fusion are achieved, improving the accuracy and robustness of target tracking.
Patent Information
- Application Number
- CN202310771396.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2025-04-18
- Estimated Expiration
- 2043-06-27
AI Technical Summary
Existing single-target trackers perform poorly when dealing with target scale changes, deformations, occlusions, and interference with similar targets, and the convolutional backbone network has limited representation capabilities, ignoring the finer structural information in the tracking.
The end-to-end single-object tracking method based on the pyramid pooling attention mechanism is adopted, and the template frames and search frames are extracted and information interaction are performed through multiple cascading pyramid pooling attention coding modules to achieve more refined feature extraction and information fusion.
This method can better capture the correlation between the target and the search area, improve the accuracy of the target tracking results, and enhance the expression ability and robustness of the model.
Smart Images
Figure CN116823885B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an end-to-end single target tracking method based on a pyramid pooling attention mechanism. Background Art
[0002] Visual object tracking, as one of the basic tasks in computer vision, aims to estimate the spatial position of any object in each frame of a video and draw the bounding box of the object. Although object tracking has made significant progress in recent years, designing a simple and efficient end-to-end tracker is still challenging. These challenges mainly come from the scale change, deformation, occlusion, and interference with similar objects of the tracked object.
[0003] The current mainstream single target tracker is mainly composed of three main modules: (1) Backbone network, which is used to extract common features of the target to be tracked and the search area; (2) Feature fusion module, which fuses the information of the tracked target and the feature information of the search area to achieve subsequent target positioning and perception; (3) Target bounding box accurate prediction module, which is used to generate the final tracking result, that is, the bounding box of the target. Among these three main modules, the core is the feature fusion module in the second part, which mainly integrates the feature information of the target to be tracked into the features of the search area, realizes the information communication between the template and the search area, guides the subsequent tracking process, and obtains the corresponding tracking box according to the specific tracking target. Traditional information fusion methods include correlation-based operations and online model update algorithms. Due to the global and dynamic modeling capabilities of Transformer, it is introduced into the tracking field to perform information interaction between the tracking target and the tracking area, which can achieve better tracking performance. It mainly uses the Transformer model to fuse the target features and the search area, and then predicts the fused features to achieve tracking. However, these Transformer-based trackers still rely on convolutional backbone networks for feature extraction, and only apply attention operations in a relatively high-level and abstract representation space. However, the representation ability of convolutional backbone networks is limited. First, it is usually pre-trained based on general target recognition tasks, and second, it may ignore more detailed structural information for tracking. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes an end-to-end single target tracking method based on a pyramid pooling attention mechanism, the method comprising: obtaining a video to be identified and preprocessing it, inputting the preprocessed image into a trained end-to-end single target tracking model, and obtaining a target tracking result;
[0005] The training process of the end-to-end single target tracking model includes:
[0006] S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame;
[0007] S2: Use a feature extractor to process the template frame and the search frame to obtain mixed features;
[0008] S3: Segment the mixed features to obtain primary search frame features; perform a Reshape operation on the search frame features to obtain the final search frame features;
[0009] S4: Use the full convolution prediction head to process the search frame features to obtain the target tracking frame, i.e., the target tracking result;
[0010] S5: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
[0011] Preferably, the process of preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is twice the target frame, and the cropped image is used as the search frame; cropping the first frame of the video frame to be identified according to the real target frame, the cropping size is four times the target frame, and the cropped image is used as the template frame.
[0012] Preferably, the process of the feature extractor processing the template frame and the search frame includes: performing convolution and splicing processing on the template frame and the search frame to obtain a splicing vector; using multiple cascaded pyramid pooling attention encoding modules to process the splicing vector to obtain a mixed feature.
[0013] Furthermore, the process of convolving and splicing the template frame and the search frame includes: adjusting the size of the template frame to 128×128×3, and adjusting the size of the search frame to 320×320×3; inputting the resized template frame and search frame into the convolution layer for processing respectively to obtain overlapping template block vectors and search block vectors; flattening and splicing the two overlapping block vectors to obtain a splicing vector; wherein the convolution kernel size of the convolution layer is 7 and the step length is 4.
[0014] Furthermore, the pyramid pooling attention encoding module processes its input features as follows:
[0015] S21: Segment the input features and perform a Reshape operation to obtain a first template feature map and a first search feature map;
[0016] S22: respectively inputting the first template feature map and the first search feature map into four global average pooling modules with different pooling ratios to obtain template pooling features and search pooling features of four different scales;
[0017] S23: Flatten and splice the template pooling features of four different scales, i.e., the multi-scale template pooling features, on the same dimension, perform linear projection processing on the spliced features to obtain the K and V vectors of the template frame; Flatten and splice the search pooling features of four different scales, i.e., the multi-scale search pooling features, splice the multi-scale search pooling features after the multi-scale template pooling features, and perform linear projection processing on the spliced features to obtain the K and V vectors of the search frame;
[0018] S24: performing attention operations on the K and V vectors of the template frame and the K and V vectors of the search frame respectively to obtain a second template feature map and a second search feature map;
[0019] S25: splicing the second template feature map and the second search feature map, and inputting the spliced features into the linear layer for processing to obtain linear features;
[0020] S26: Add the linear features and the input features to obtain the module output features.
[0021] Preferably, the process of obtaining the target tracking frame includes: using 5 convolutional layers to process the search frame features to obtain the upper left corner heat map and the upper right corner heat map of the target; taking the maximum values of the two heat maps as the upper left corner coordinates and the upper right corner coordinates of the target respectively; generating the target tracking frame according to the upper left corner coordinates and the upper right corner coordinates of the target.
[0022] Preferably, the total loss of the model is:
[0023]
[0024] Among them, L track Represents the total loss value of the model, L cls represents the classification error, L IoU represents the IoU loss between the predicted box and the true box, λ IoU Indicates L IoU The weight value of , L1 represents the bounding box regression loss, Represents the weight value of L1.
[0025] The beneficial effects of the present invention are as follows: the present invention repeats the pyramid pooling attention module multiple times to obtain a feature extractor based on the module, and deepens the network depth through multiple such serial modules. The module simultaneously performs feature extraction and information interaction of the target template and the search area. In the pyramid pooling attention module, the present invention designs a hybrid interaction scheme to perform self-attention and cross-attention operations on the features from the target template and the search area, that is, the template frame and the search frame. Self-attention is responsible for extracting the own features of the target template or the search area, while cross-attention ensures communication between them to fuse the target and search area information. In addition, the present invention also uses pooling to fuse feature information of different scales, which reduces the amount of calculation on the one hand, and also learns rich scale representations, making full use of the feature modeling capabilities of the Transformer, and enhancing the expressiveness and robustness of the model. The present invention can better capture the correlation between the target and the search area and improve the accuracy of the target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a flow chart of end-to-end single target tracking model training in the present invention;
[0027] Figure 2 It is a specific implementation flow chart of the pyramid pooling attention encoding module in the present invention;
[0028] Figure 3 It is a schematic diagram of the structure of the prediction head in the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] This paper proposes an end-to-end single target tracking method based on pyramid pooling attention mechanism. Figure 1 As shown, the method includes the following contents: obtaining a video to be identified and preprocessing it, inputting the preprocessed image into a trained end-to-end single target tracking model to obtain a target tracking result; the end-to-end single target tracking model (P2TT) includes a feature extractor (backbone network) and a fully convolutional prediction head (tracking head).
[0031] The training process of the end-to-end single target tracking model includes:
[0032] S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame.
[0033] The process of obtaining the video to be identified for training and preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is twice the target frame, and the cropped image is used as the search frame; cropping the first frame of the video frame to be identified according to the real target frame, the cropping size is four times the target frame, and the cropped image is used as the template frame. Among them, the label of the video to be identified for training is the coordinates of the upper left corner and the lower right corner of the real target frame.
[0034] S2: Use the feature extractor to process the template frame and the search frame to obtain mixed features.
[0035] This feature extractor is based on the pyramid pooling attention mechanism. Different from the traditional feature extractor, this feature extractor makes full use of the characteristics of Transformer. While extracting features for target tracking, it also interacts with the information of the template and the search area to effectively model the tracking. The process of using the feature extractor to process the template frame and the search frame is as follows:
[0036] The template frame and the search frame are convolved and concatenated to obtain a concatenated vector. Specifically:
[0037] The template frame is resized to 128×128×3, and the search frame is resized to 320×320×3. The resized template frame and search frame are respectively input into the convolutional layer for processing to obtain overlapping template block vectors and search block vectors. The two overlapping block vectors are flattened and concatenated to obtain the concatenated vector F. token ; Preferably, the convolution kernel size of the convolution layer is 7 and the step length is 4.
[0038] Multiple cascaded pyramid pooling attention encoding modules are used to process the concatenated vectors to obtain mixed features. Figure 2 As shown in Figure 2, the pyramid pooling attention encoding module processes its input features in the following way:
[0039] S21: Segment the input features according to the flattened feature size corresponding to the template frame or the search frame and perform a Reshape operation to obtain a first template feature map and a first search feature map.
[0040] S22: Input the first template feature map and the first search feature map after the Reshape operation into four global average pooling modules with different pooling ratios respectively, to obtain template pooling features and search pooling features of four different scales.
[0041] S23: Flatten and splice four template pooling features of different scales on the same dimension (called "multi-scale template pooling features"), perform linear projection on the spliced features to obtain the K and V vectors of the template frame; Flatten and splice four search pooling features of different scales (called "multi-scale search pooling features"), splice the multi-scale search pooling features after the multi-scale template pooling features, and perform linear projection on the spliced features to obtain the K and V vectors of the search frame;
[0042] S24: Perform attention operations on the K and V vectors of the template frame and the K and V vectors of the search frame respectively to obtain a second template feature map and a second search feature map.
[0043] Specifically, for the first template feature map, a self-attention operation is used to extract feature information. The Q in the attention comes from the linear projection mapping of the input template feature map, and K and V come from the linear projection mapping of the multi-scale template pooling features, and the parameters are shared. For the first search feature map, a cross-attention operation is used to complete feature extraction and fusion template feature information. Its Q comes from the linear projection mapping of the first search feature map, and K and V come from the linear projection of the spliced multi-scale template pooling features and the multi-scale search pooling features, and the parameters are shared.
[0044] S25: splicing the second template feature map and the second search feature map, and inputting the spliced features into the linear layer for processing to obtain linear features;
[0045] S26: Add the linear features and the input features to obtain the module output features.
[0046] The concatenated vector is input to multiple cascaded pyramid pooling attention encoding modules to repeat the operations S21-S26 multiple times to obtain a feature vector that has been deeply coded and fused, i.e., a mixed feature; it combines the template frame information (tracking target information) and the search frame information.
[0047] S3: Segment the mixed features to obtain the primary search frame features; perform a Reshape operation on the primary search frame features to obtain the final search frame features F test .
[0048] The mixed features are split according to the flattened feature sizes of the template frame and the search frame, the search features are taken out, and after the reshape operation, they are used as the input of the convolution prediction head. The final search frame feature F test The size is 20×20×384.
[0049] S4: Use the full convolution prediction head to process the search frame features to obtain the target tracking frame, i.e., the target tracking result.
[0050] like Figure 3As shown in the figure, the full convolution prediction head includes a classification head and a regression head; the process of the regression head processing the search frame features includes: using 5 convolutional layers to process the search frame features, the number of input channels is 384, the number of output channels is 2, and the upper left corner heat map and the upper right corner heat map of the target are obtained, each with a size of 20×20×1; the maximum values of the two heat maps are taken as the upper left corner coordinates and the upper right corner coordinates of the target; the target tracking frame is generated according to the upper left corner coordinates and the upper right corner coordinates of the target.
[0051] The process of the classification head processing the search frame features includes:
[0052] The encoded search frame features are used as the input of the classification head. After stacking convolution layers, linear projection layers and a Softmax operation, the probability that the target is the foreground is obtained, that is, the confidence of the target tracking box.
[0053] S5: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
[0054] The process of calculating the total loss of the model includes:
[0055]
[0056] Where L track Represents the total loss value of the model during training, L cls Represents the classification error, calculated using Focal loss, L IoU is the IoU (intersection over union) loss between the predicted box and the real box, λ IoU For L IoU The weight value is set to 2, L1 is the bounding box regression loss, is the weight value of L1, which is set to 5. The specific algorithms of each loss function are as follows:
[0057] L cls =-[t*log(sigmoid(p))+(1-t)*log(1-sigmoid(p))]
[0058] Among them, t is the true label, 0 represents the target, 1 represents the background; p represents the probability of being predicted as the target.
[0059]
[0060] Among them, Bbox pre Indicates the predicted target position coordinate information, Bbox Gt The real coordinate information of the target.
[0061] L1=|x p -x t|+|y p -y t |+|w p -w t |+|h p -h t |
[0062] Among them, (x p ,y p ,w p ,h p ) is the predicted bounding box, (x t ,y t ,w t ,h t ) is the true bounding box.
[0063] During the training process, the AdamW optimizer can be used, the single-card BatchSize is set to 32, the total number of training rounds is set to 500, the learning rate is divided by 10 at 400 rounds, and training is performed on 4 Nvidia GeForce RTX 3090s. The entire network parameters are updated through the back-propagation algorithm.
[0064] After obtaining the trained end-to-end single target tracking model; obtain the video to be identified and preprocess it, and input the preprocessed image into the trained end-to-end single target tracking model to obtain the target tracking result.
[0065] The end-to-end single target tracking model designed by the present invention simultaneously performs feature extraction and information fusion by adopting a feature extractor and a fully convolutional prediction head. The feature extractor designed by the present invention is more suitable for specific tracking targets and captures more discriminative features related to the target. At the same time, it also allows the fusion of target information of more scales, thereby better capturing the correlation between the target and the search area, and improving the accuracy of the target tracking results.
[0066] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An end-to-end single target tracking method based on pyramid pooling attention mechanism, characterized in that: include: Obtain the video to be identified and preprocess it, and input the preprocessed image into the trained end-to-end single target tracking model to obtain the target tracking result; The training process of the end-to-end single target tracking model includes: S1: Obtain the video to be identified and preprocess it to obtain a template frame and a search frame; S2: The template frame and the search frame are processed by a feature extractor to obtain a mixed feature; the process of the feature extractor processing the template frame and the search frame includes: performing convolution and splicing processing on the template frame and the search frame to obtain a splicing vector; using multiple cascaded pyramid pooling attention encoding modules to process the splicing vector to obtain a mixed feature; the process of the pyramid pooling attention encoding module processing its input feature includes: S21: Segment the input features and perform a Reshape operation to obtain a first template feature map and a first search feature map; S22: respectively inputting the first template feature map and the first search feature map into four global average pooling modules with different pooling ratios to obtain template pooling features and search pooling features of four different scales; S23: Flatten and splice the template pooling features of four different scales, i.e., the multi-scale template pooling features, on the same dimension, perform linear projection processing on the spliced features to obtain the K and V vectors of the template frame; Flatten and splice the search pooling features of four different scales, i.e., the multi-scale search pooling features, splice the multi-scale search pooling features after the multi-scale template pooling features, and perform linear projection processing on the spliced features to obtain the K and V vectors of the search frame; S24: performing attention operations on the K and V vectors of the template frame and the K and V vectors of the search frame respectively to obtain a second template feature map and a second search feature map; S25: splicing the second template feature map and the second search feature map, and inputting the spliced features into the linear layer for processing to obtain linear features; S26: Add the linear feature and the input feature to obtain the module output feature; S3: Segment the mixed features to obtain primary search frame features; perform a Reshape operation on the primary search frame features to obtain the final search frame features; S4: Use the full convolution prediction head to process the search frame features to obtain the target tracking frame, i.e., the target tracking result; S5: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained end-to-end single target tracking model.
2. According to claim 1, an end-to-end single target tracking method based on pyramid pooling attention mechanism is characterized in that: include: The process of preprocessing the video to be identified includes: cropping the video frame to be identified according to the center position of the real target frame, the cropping size is twice the target frame, and the cropped image is used as the search frame; The first frame of the video frame to be recognized is cropped according to the real target frame, and the cropping size is 4 times that of the target frame. The cropped image is used as the template frame.
3. The end-to-end single target tracking method based on pyramid pooling attention mechanism according to claim 1, characterized in that: The process of convolution and splicing the template frame and the search frame includes: adjusting the size of the template frame to 128×128×3, and adjusting the size of the search frame to 320×320×3; inputting the resized template frame and search frame into the convolution layer for processing respectively to obtain overlapping template block vectors and search block vectors; flattening and splicing the two overlapping block vectors to obtain a splicing vector; wherein the convolution kernel size of the convolution layer is 7 and the step length is 4.
4. The end-to-end single target tracking method based on pyramid pooling attention mechanism according to claim 1, characterized in that: The process of obtaining the target tracking frame includes: using 5 convolutional layers to process the search frame features to obtain the upper left corner heat map and the upper right corner heat map of the target; taking the maximum values of the two heat maps as the upper left corner coordinates and the upper right corner coordinates of the target respectively; generating the target tracking frame according to the upper left corner coordinates and the upper right corner coordinates of the target.
5. The end-to-end single target tracking method based on pyramid pooling attention mechanism according to claim 1, characterized in that: The total loss of the model is: Among them, L track Represents the total loss value of the model, L cls represents the classification error, L IoU represents the IoU loss between the predicted box and the true box, λ IoU Indicates L IoU The weight value of L1 represents the bounding box regression loss, Represents the weight value of L1.