An optical flow compensation-based target tracking network structure and a tracking method
By using a target tracking network structure based on optical flow compensation, the problem of target tracking performance degradation under occlusion and illumination changes is solved, achieving higher detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2024-04-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing target tracking algorithms suffer from performance degradation in complex scenes, especially under occlusion and lighting changes, making it difficult to effectively track targets.
A target tracking network structure based on optical flow compensation is adopted, including a Siamese network, an optical flow-based feature compensation module, and a Transformer feature fusion module. Through local and global occlusion compensation, feature robustness is improved.
Significantly improves the accuracy and robustness of object detection in occluded environments, with an average accuracy improvement of 0.7% on the CrowdHuman dataset, a false positive rate reduction of 0.6% per image, and a 1% improvement in the Yaca index.
Smart Images
Figure CN118552584B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and relates to a target tracking network structure and tracking method based on optical flow compensation. Background Technology
[0002] During object imaging, numerous environmental factors can affect the appearance of a scene to varying degrees, including differences in light intensity, incident light angle, shadow occlusion, and pose changes. These changes will ultimately be reflected in the image. For the same target object, different images will be captured under different environmental conditions, thus posing certain challenges to subsequent image understanding tasks.
[0003] Object tracking, a hot topic in computer vision, has been successfully applied in multiple fields. It involves a wide range of disciplines, including pattern recognition, image processing, computer science, and mathematical statistics. While existing object tracking algorithms perform excellently on classic datasets, their performance degrades in complex scenes, such as those with significant occlusion or varying lighting conditions. Researching object tracking in occluded scenarios not only advances computer vision research on this problem but also has significant practical implications for the successful application of object tracking technology in both civilian and military settings.
[0004] Target tracking methods can be mainly divided into two categories: the first is based on correlation filtering, and the second is based on deep learning.
[0005] Most correlation filtering-based algorithms use filter templates to record and update the location information of the tracked target. They then calculate the similarity between the template image and each location in each frame, using this similarity as the output response map. The location with the highest response value in the output response map is the most similar location, which is also the target's location. Deep learning-based methods use deep neural networks to extract image features. In real life, image recognition tasks become more challenging due to complexities such as occlusion, lighting variations, and target deformation. However, deep learning-based target tracking algorithms, with their superior feature extraction capabilities, can effectively address these challenges. Compared to correlation filtering-based algorithms, deep learning-based methods can better generalize to these complexities, and therefore have become the mainstream research direction. Summary of the Invention
[0006] Technical problems to be solved
[0007] To avoid the shortcomings of existing technologies, this invention proposes a target tracking network structure and tracking method based on optical flow compensation. In response to the feature damage problem that occurs in occlusion, an innovative feature compensation module based on optical flow is proposed, which enables local and global compensation of search image features, and ultimately effectively solves the impact of occlusion.
[0008] Technical solution
[0009] A target tracking network structure based on optical flow compensation is characterized by comprising a backbone network in the form of a twin network, an optical flow-based feature compensation module and a Transformer feature fusion module, as well as a classification head and a regression head;
[0010] The inputs to the backbone network in the form of the twin network are the template image and the search image, respectively. The output of the backbone network of the template image is connected to the first self-attention mechanism module of the Transformer feature fusion module, and is also connected to the optical flow-based feature compensation module. The output of the backbone network of the search image is connected to the optical flow-based feature compensation module.
[0011] The Transformer feature fusion module enhances features in different ways, based on a self-attention context enhancement module and a cross-attention mechanism feature enhancement module. It includes two self-attention mechanism modules and three cross-attention mechanism modules. The first self-attention mechanism module is connected to the first and second cross-attention mechanism modules, the second self-attention module is connected to the first and second cross-attention mechanism modules, the outputs of the first and second cross-attention mechanism modules are connected to the third cross-attention mechanism module, and the output of the third cross-attention mechanism module is connected to the classification head and the regression head.
[0012] The optical flow-based compensation module performs feature compensation in different ways, including local occlusion compensation and semantic-based global occlusion compensation. It includes a local compensation module and a global compensation module. The output of the local compensation module is connected to the global compensation module, and the output of the global compensation module is connected to the Transformer feature fusion module.
[0013] The output of the search image backbone network is connected to the optical flow-based feature compensation module, and then to the second self-attention mechanism module of the Transformer feature fusion module.
[0014] The backbone network uses an improved deep neural network, ResNet50, which changes the stride of the fourth downsampling unit from 2 to 1; and changes the 3×3 convolution in the fourth layer to a dilated convolution with a stride of 2 to obtain a larger receptive field. The backbone network retains the ResNet50 network structure except for the fully connected layers, and is used to learn and extract the inherent features of the image.
[0015] The optical flow-based feature compensation module includes a local occlusion compensation module and a semantic-based global compensation module, which are used to compensate for image features damaged by occlusion in the search image.
[0016] The backbone networks of the template image and the search image are weight-shared backbone networks.
[0017] The Transformer feature fusion module includes a context enhancement module based on self-attention and a feature enhancement module based on cross-attention mechanism, which are used to capture similarity information between the search image and the template image.
[0018] The target tracking network structure is trained using a training image sample library. The training image sample library is constructed by randomly selecting the required number of images from the training image dataset, and then randomly selecting a region in each image to generate an occlusion mask. The occlusion images obtained after performing the above operation on all the selected images constitute the training image sample library.
[0019] The target tracking network structure is trained using samples from a training image sample library. The Adam optimization method is used to optimize the network, with the optimizer parameters betas set to (0.5, 0.999), the learning rate to 0.0005, the weight_decay to 0.0001, and the batch size to 20.
[0020] A method for target tracking using the aforementioned target tracking network structure based on optical flow compensation, characterized by the following target tracking steps:
[0021] Step 1: Extract the image features of the template image and the search image through their respective backbone networks;
[0022] Step 2: The template image features and search image features are fed into the optical flow-based feature compensation module. The two features are first input into the local occlusion compensation module, which calculates the forward and backward optical flow between the template image features and the search image features. The obtained forward and backward optical flows are then checked for consistency to determine the occlusion region, thereby generating an occlusion mask. The calculation formula is as follows:
[0023]
[0024] Where x represents the coordinates of the pixel, w f (x) represents the forward warp operation based on the calculated optical flow, w b This indicates that a backward mapping operation is performed on the optical flow, where α1 and α2 are two adjustment parameters. In the case of no occlusion, the equation on the left side tends to 0. If the equation is not satisfied, it can be considered that there is occlusion. According to the equation, a mask for the occluded area can be calculated. When the corresponding pixel is occluded, the value of the corresponding pixel on the mask should be 1, otherwise it should be 0.
[0025] Template image features are incorporated into the occluded regions of the search image features to obtain locally compensated search image features. The calculation formula is as follows:
[0026] F ps =F t *mask+F s *(1-mask)
[0027] Among them, F s To search for image features, F t For template image features, F ps The search image features are those after local feature compensation;
[0028] Step 3: The results of Step 2 and the template image features are processed by the global compensation module. First, the optical flow of the template image features and the locally compensated search image features is calculated. The template image features are mapped by optical flow. Then, an attention mask is used to represent the weight of important regions of the search image. The mask is used to fuse the mapped template image features and the locally compensated search image features to obtain the globally compensated search image features. The calculation formula is as follows:
[0029] F os =mask_p*F ps +(1-mask_p)*F wt
[0030] Among them, F wt F represents the mapped template features. os The search image features represent global compensation, and mask_p represents the weight mask for important regions of the search image;
[0031] Step 4: The results of Step 3, namely the search image features with global feature compensation and the template image features, are fed into the first self-attention mechanism module through the Transformer feature fusion module. The calculation formula is as follows:
[0032]
[0033] Among them, Q, K, and V are generated by mapping the features of the template image;
[0034] The output of the first self-attention mechanism is fed into both the first and second cross-attention mechanisms. The globally compensated search image features are fed into the second self-attention mechanism. The calculation method of the second self-attention mechanism is the same as that of the first self-attention mechanism, only the input is different. The output of the second self-attention mechanism is fed into both the first and second cross-attention mechanisms. The calculation formula of the first cross-attention mechanism is the same as that of the first self-attention mechanism, except that Q is generated by mapping the output of the first self-attention mechanism, and K and V are generated by mapping the output of the second self-attention mechanism. The Q of the second cross-attention mechanism is generated by mapping the output of the second self-attention mechanism, and K and V are generated by mapping the output of the first self-attention mechanism. The input of the first cross-attention mechanism and the output of the second cross-attention mechanism are fed into the third cross-attention mechanism. The output of the third cross-attention mechanism is fed into the regression head and the classification head.
[0035] Step 5: The results of Step 4 are fed into the regression head and the classification head to obtain the final tracking result, i.e., the target's position coordinates.
[0036] The specific process of obtaining tracking results using the classification head and regression head in step 5 is as follows:
[0037] Step 5.1: Perform mean subtraction preprocessing on the original template image and search image, adjust the size of the template image to 128*128, and adjust the size of the search image to 256*256. When the size exceeds the original image size, fill with the average RGB value; in order to avoid adjusting the image size during training, the template image and search image of each frame are extracted offline.
[0038] Step 5.2: A fine-tuned network training method is adopted. After the images are subtracted from their mean and resized, the template image and the search image are fed into the target tracking network for training. During fine-tuning, the loss function is defined as the cross-entropy function between pixels and the distance function between target locations. The classification result given by the classification head is compared with the corresponding annotations of the image at the pixel level. The classification loss function is:
[0039]
[0040] Among them, y j =1 indicates that this pixel is a positive sample, p j This represents the probability that the model predicts it to be a positive sample; the target location result given by the regression head is compared with the corresponding labeled location in the image, and the regression loss function is:
[0041]
[0042] This includes L1 loss and GIoU loss; y j =1 represents a positive sample, b j This represents the target bounding box predicted by the tracking model. Then, λ represents the normalized predicted target bounding box. G Set to 2, λ1 is set to 5;
[0043] Tracking loss function L occ and L ori The calculation formula for L: track =L cls +L reg .
[0044] Step 5.3: Perform several iterations of training on each pair of search images and template images, and judge the convergence of the model by the change of the loss function value until the model reaches the convergence state.
[0045] A computer program product characterized by comprising computer-executable instructions, which, when executed, are used to implement the method described.
[0046] Beneficial effects
[0047] This invention proposes a target tracking network structure and method based on optical flow compensation. For target detection in occluded environments, it proposes an occluded target detection method based on local perception. Specifically, at the feature level, a local perception module is proposed. Multiple local perception branches explore multiple local regions and use a random discarding strategy to force other branches to learn important local information from salient features. Considering the problem that the detector may predict multiple candidate detection boxes, a query update module is proposed. This module uses the spatial location information of the target to remove excessive candidate boxes, improving the accuracy of the detection results. Finally, on the CrowdHuman dataset, the proposed algorithm achieves a 0.7% improvement in average accuracy compared to RT-DETR, a 0.6% gain in the log-mean false positive rate per image, and a 1% gain in the Yaca index. Attached Figure Description
[0048] Figure 1 Network overall framework diagram
[0049] Figure 2 Illustrations of two residual structures
[0050] Figure 3 Schematic diagram of the partial occlusion compensation module
[0051] Figure 4 Schematic diagram of the global feature compensation module
[0052] Figure 5 Diagram of self-attention
[0053] Figure 6 Diagram of cross-attention
[0054] Figure 7 Heatmap of global feature compensation module
[0055] Figure 8 Heatmap of the partial occlusion compensation module
[0056] Figure 9 Success rate and accuracy plots for various target tracking algorithms on the OTB-occlusion dataset. Detailed Implementation
[0057] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:
[0058] This invention explores and proposes a novel target tracking algorithm based on optical flow compensation. In terms of the backbone network, based on the ResNet50 algorithm, this invention changes the stride of the fourth-layer downsampling unit from 2 to 1, increasing the network's receptive field. Regarding the optical flow-based feature compensation module, this invention innovatively proposes local and global feature compensation. After performing local occlusion compensation and global semantic compensation on the backbone network features, a global feature fusion combining temporal template features and current features is obtained and fed into the Transformer feature fusion module. This module repairs damaged features and improves tracking performance. In the Transformer feature fusion module, this invention incorporates a combination of self-attention and cross-attention modules. On one hand, a context enhancement module based on self-attention establishes long-distance dependencies, ensuring that each feature point focuses on useful information in the image, thereby enhancing the image's feature representation. On the other hand, a residual structure based on cross-attention enhances the feature representation of the template image and the search image. This module uses template image features and search image features as input, thus not only establishing a global relationship between the template image and the search image but also capturing the correlation information between the search image and the template image. A feature fusion module that combines self-attention and cross-attention is used to capture global aggregated features that are more beneficial for tracking tasks.
[0059] To achieve the above objectives, this invention proposes a target tracking network and method based on optical flow compensation, the network structure of which is as follows: Figure 1 As shown, the network structure and target tracking steps are as follows:
[0060] Step 1: Construct a training image sample library
[0061] (1) Constructing a training image sample library
[0062] This invention randomly selects a certain number of images from the training image dataset, and then randomly selects a region in each image to generate an occlusion mask; the occlusion images obtained after performing the above operation on all the selected images constitute the training image sample library.
[0063] In this example, the LaSOT and GOT-10K object tracking datasets were used. The LaSOT dataset contains 1400 video segments totaling 3,500,000 frames, with each video sequence containing an average of 2500 frames. Of these 1400 video sequences, 511 were designated as test sequences, and the rest were used as training data. The LaSOT video sequences are all derived from real-life events, a characteristic well-suited for single-object tracking tasks. The GOT-10K dataset, like LaSOT, is a large-scale dataset and currently the most comprehensive single-object tracking dataset in terms of object categories. GOT-10K includes 87 sports and 563 categories, such as tennis, cycling, running, and soccer. This dataset contains 10180 video segments, of which 180 are test sequences and 10000 are training data.
[0064] This invention further improves upon image occlusion generation by employing an incremental occlusion generation module. In this module, the increment refers to the fact that the size of the generated occlusion region is not constant. The size of the occlusion region increases with each training iteration. This invention achieves a progression from easy to difficult training data by gradually increasing the size of the occlusion region. The algorithm learns from easy data first, understanding how to handle simple occlusion situations, and then gradually increases the occlusion region to expose the algorithm to more severe and challenging data. This gradual increase in difficulty enhances the algorithm's ability to solve occlusion problems. If the network were exposed to the most difficult occlusion data from the beginning, it might not initially learn effective information from this data; a progression from easy to difficult aligns better with learning principles. Furthermore, the algorithm does not use a manually predetermined strategy for the location of the generated occlusion region in the image; it randomly selects the location. This randomness in selecting the occlusion region also means that the target may not be occluded, thus encompassing both occluded and unoccluded scenarios.
[0065] Step 2: Construct the target tracking network
[0066] according to Figure 1 , Figure 2 , Figure 3 , Figure 4 as well as Figure 5The target tracking network is constructed using the method in step 2. The search image and template image are processed through a backbone network to obtain their image features. These features are then fed into an optical flow-based feature compensation module to obtain locally and globally compensated search image features. Finally, the compensated search image features and template image features are fed into a Transformer feature fusion module for feature fusion.
[0067] (1) Constructing a backbone network
[0068] The backbone network is modeled after the ResNet50 network structure, but the stride of the downsampling unit in the fourth layer of ResNet50 is changed from 2 to 1 to increase the receptive field. Furthermore, the 3×3 convolutions in the fourth convolutional layer of ResNet50 are replaced with dilated convolutions with a stride of 2 to achieve an even larger receptive field. The backbone network employs two different residual structures, such as... Figure 2 As shown, the backbone network consists of five convolutional layers: conv1, conv2.x, conv3.x, conv4.x, and conv5.x. Except for conv1, all other convolutional layers in the backbone network are composed of multiple basic residual modules. The convolutional units conv2.x, conv3.x, conv4.x, and conv5.x are stacked with 3, 4, 6, and 3 basic residual modules respectively.
[0069] (2) Constructing a feature compensation module based on optical flow
[0070] For the partial occlusion compensation module, its structural diagram is as follows: Figure 3 As shown. This module is for determining the occlusion area. This paper proposes to adopt the optical flow consistency criterion. The optical flow consistency criterion means that if there is no occlusion, the forward-mapped optical flow vector can be restored to its original position when it is mapped backward. If there is occlusion, such as a person walking on the street being blocked by objects such as trees or cars, the part of the pedestrian that is blocked cannot be found in the next frame. Therefore, the criterion for determining whether there is occlusion is that if the current optical flow cannot return to its original position after forward and backward mapping, it can be identified as occlusion, as shown in Equation (1):
[0071]
[0072] Where x represents the coordinates of the pixel, w f (x) represents the forward warp operation based on the calculated optical flow, w bThis indicates a backward mapping operation on the optical flow, where α1 and α2 are two adjustment parameters. In the absence of occlusion, the equation on the left side of equation (1) tends to 0. If equation (1) is not satisfied, then occlusion is considered to exist. According to equation (1), a mask for the occluded region can be calculated. When the corresponding pixel is occluded, the corresponding pixel value on the mask should be 1, otherwise it should be 0. In this module, the forward and backward optical flow between the search image features and the template image features are first calculated. The forward and backward optical flow are used to perform an optical flow consistency check to determine the occluded region, and finally, an occlusion mask is generated. Then, the occlusion mask is used to fuse the search image features of the occluded region with the template image features to repair the features of the occluded region to a certain extent, and finally, a search image feature with local feature compensation is obtained.
[0073] For the global feature compensation module, its structural diagram is as follows: Figure 4 As shown in the diagram. This module first maps and matches the occlusion-compensated search features with the template features to obtain template features semantically aligned with the search features, such as... Figure 4 As shown. Then, an attention module is used to capture the information that needs to be focused on in the search image, so as to maintain the original level of attention to the important regions of the search image features. At the same time, the temporal information of the template features can be used to highlight some neglected image regions. Figure 4 The attention mask in the model is used to fuse semantically aligned template images with occlusion-compensated search features to obtain global features that integrate temporal template features and current search features. This invention proposes using semantically aligned template images to compensate for features in the search image, ensuring that some potentially important regions in the search image features are addressed. This module aims to preserve information that should already be emphasized in the search image, while utilizing template images to enrich temporal information, thereby uncovering more valuable regions.
[0074] (3) Construct a feature fusion module based on Transformer
[0075] This module aims to capture the correlation information between the search image and the template image through feature fusion, which is used for feature fusion between the search image and the template image. This module uses Transformer to construct a context enhancement module based on self-attention and a feature fusion module based on cross-attention mechanism; both modules perform feature enhancement in different ways.
[0076] For the self-attention-based context enhancement module, its structural diagram is as follows: Figure 5As shown, this module is a residual structure based on a self-attention mechanism. This module establishes long-distance dependencies through the image's own self-attention mechanism, ensuring that each feature point focuses on useful information in the image, thereby enhancing the image's feature representation. For the search image, this module first stretches the search image features fed into the backbone network into long vectors for use by the Transformer module. Then, it uses the search image vector mapping to generate the query matrix Q, key matrix K, and value matrix V required by the Transformer.
[0077] For the feature enhancement module based on cross-attention, its structural diagram is as follows: Figure 6 As shown, this module is a residual structure based on a multi-head cross-attention mechanism. For the cross-attention feature enhancement module of the search image, it introduces template image features as another input in addition to itself as input. This module performs cross-attention fusion of the two features through the Transformer's cross-attention mechanism, which can perform global and non-linear feature fusion, capturing global information of features while retaining a large amount of rich semantic information. Compared with the context enhancement module based on self-attention, the query matrix Q and key matrix K of this module do not come from the same object, while the query matrix Q and key matrix K of the context enhancement module based on self-attention come from the same object.
[0078] The entire Transformer-based feature fusion module comprises two context enhancement modules based on self-attention and two feature enhancement modules based on cross-attention. These two modules, along with the cross-attention module, form a feature fusion layer. The feature fusion loop is performed N times, and the final fused features are decoded by a cross-attention feature enhancement module. The entire feature fusion module captures globally aggregated features that are more beneficial for target tracking tasks through multiple self-feature enhancements and cross-feature enhancements.
[0079] Step 3: Define the loss function
[0080] According to formulas (2)-(5), the generator loss function is calculated in this invention.
[0081] When calculating the classification loss, this invention forces the network to update its own parameters so that the tracking head can accurately identify positive and negative samples in the image.
[0082] In order to better reflect the degree of overlap between any two boxes when calculating the regression loss, this invention uses GiOU loss.
[0083] (1) Define the target tracking network loss as shown in formula (2):
[0084] L all =αL occ +βL ori (2)
[0085] Among them, L occ L represents the tracking loss obtained by using the occluded search image and the template image. ori The α and β values represent the tracking loss obtained using the original search image and the template image, respectively. This invention generates a search image affected by occlusion using an incremental occlusion generation module. During training, the original search image is not discarded; instead, the occluded search image, the original search image, and the template image are fed together into the tracking network. That is, the network input consists of two search images and one template image. This training design aims to ensure that training on occluded data does not degrade the tracking performance of the algorithm in normal scenes. Therefore, the occluded search image and the template image are included in the training. A set of classification and regression losses can be obtained using the occluded search image and the template image, and a set of classification and regression losses can also be obtained using the original search image and the template image.
[0086] Tracking loss function L occ and L ori The calculation formula is shown in formula (3):
[0087] L track =L cls +L reg (3)
[0088] Among them, L track Let L represent the tracking loss function. cls L represents the classification loss. reg This represents the regression loss.
[0089] Classification loss function L cls The calculation formula is shown in formula (4):
[0090] L cls =-∑ j [y j log(p j )+(1-y j log(1-p) j (4)
[0091] This loss is the cross-entropy loss, and both positive and negative samples participate in its calculation. Where y j =1 indicates that this pixel is a positive sample, p jThis represents the probability that the model predicts it to be a positive sample. To minimize the loss, the tracking network needs to be forced to update its own parameters so that the tracking head can accurately distinguish between positive and negative samples in the image.
[0092] Regression loss function L reg The calculation formula is shown in formula (5):
[0093]
[0094] The regression loss consists of L1 loss and GIoU loss, and only positive samples are included in the regression loss calculation. Where y j =1 represents a positive sample, b j This represents the target bounding box predicted by the tracking model. Then, λ represents the normalized predicted target bounding box. G Set it to 2, and λ1 to 5.
[0095] Step 4: Network Training
[0096] The target tracking network in step 2 is trained using the training image sample library constructed in step 1. The network parameters are updated using the Adam optimization method. Training stops when the loss function value defined in step 3 is minimized, and the final target tracking network is obtained.
[0097] The optimizer parameters betas are set to (0.5, 0.999), the learning rate is 0.0005, the weight_decay is 0.0001, and the batch size is 20.
[0098] Step 5: Input the original image to be processed into the occlusion image obtained in Step 1 and send it into the target tracking network. The output result is the tracking result.
[0099] To comprehensively evaluate the effectiveness of the algorithm proposed in this invention, this invention first analyzes the effect of the optical flow feature compensation module, and then compares and evaluates it on the OTB-occlusion target tracking dataset and the VOT-occlusion target tracking dataset through two aspects: visualization effect and quantitative indicators.
[0100] (1) Evaluation of the global feature compensation module
[0101] To evaluate the effectiveness of the global feature compensation module, this invention uses the VOT-occlusion dataset for evaluation. A set of comparative experiments is designed to evaluate the global feature compensation module.
[0102] (a) Use optical flow to compensate for the global feature module of the search image.
[0103] (b) Optical flow is not used to compensate for global features of the search image.
[0104] With other variables remaining constant, this invention trains the two networks mentioned above, and the final results on the VOT-occlusion dataset are shown in Table 1.
[0105] Table 1 Comparison of those with and without a global feature compensation module (those marked in black are the best).
[0106]
[0107] As shown in Table 1, although there are some shortcomings in robustness, using optical flow to compensate for the global features of the search frame will result in better overall performance and better average accuracy. This also shows that using optical flow to supplement the global features of the search frame can maintain good accuracy across the entire dataset and will not cause the tracking algorithm to have a certain bias in certain scenarios. Figure 7 A heatmap of the global feature compensation module is shown, generated using Grad-CAM. Figure 7 It can be observed that after global feature compensation, the attention is not only focused on the boundary region of the target, but also covers the target itself. For example, in Figure 3-6 In this study, the features not only focused on the boy's face obscured by the magazine, but also on the uncovered portion of his face. This focus on both obscured and uncovered areas helps to obtain more robust features, thereby improving tracking performance.
[0108] (2) Evaluation based on local occlusion compensation module
[0109] To evaluate the effectiveness of the local occlusion compensation module, this invention uses the VOT-occlusion dataset for evaluation. A set of comparative experiments was designed to evaluate the local occlusion compensation module.
[0110] (a) Optical flow was used to compensate for the occlusion features of the search image.
[0111] (b) Instead of using optical flow to compensate for the features of the occluded areas of the search image, optical flow is used directly to compensate for the global features of the image.
[0112] With other variables remaining constant, this invention trains the two networks mentioned above, and the final results on the VOT-occlusion dataset are shown in Table 2.
[0113] Table 2 Comparison of modules with and without local occlusion compensation (those marked in black are the best).
[0114]
[0115] As shown in Table 2, scheme a, which uses optical flow to compensate for the features of occluded regions in the search image, outperforms scheme b, which does not, in all metrics of this dataset. Scheme a not only surpasses scheme b in average accuracy and overall performance but also in tracking robustness. This comparative experiment demonstrates the effectiveness of using optical flow for feature compensation in the search image. Feature compensation for occluded regions in the search image enhances its feature representation, supplementing the feature representation of the occluded areas. The compensated search image features are more helpful for the model to predict the final tracking results. Figure 8 A heatmap of the local occlusion compensation module is shown, generated using Grad-CAM. Figure 8 As can be seen, after local occlusion compensation, the features focus on the boundary region of the target, and important regions are not ignored due to occlusion. For example, in Figure 8 In the photo, although the boy's face is obscured by the magazine, his features are still able to be focused on the obscured facial area.
[0116] Based on the above indicators and visualization results, it can be concluded that the optical flow-based feature compensation proposed in this invention has a significant effect on solving the occlusion problem in target tracking tasks, and significantly improves image robustness and tracking accuracy. After the optical flow-based feature compensation module is activated, this invention can effectively restore image features lost due to occlusion.
[0117] (2) Analysis of experimental results of OTB-occlusion dataset
[0118] To verify the universality of the proposed method on different datasets, this invention also tested it on the OTB-occlusion dataset. This invention selected a subset of the OTB100 dataset with occlusion labels as the test set, and the specific quantitative indicators are shown in Table 3. It can be seen that the proposed method also achieved the best results on the OTB-occlusion dataset, proving the effectiveness of this method. Figure 9 Success rate and accuracy plots for various target tracking algorithms on the OTB-occlusion dataset.
[0119] Table 3 Comparison of tracking results of various tracking algorithms on the OTB-occlusion dataset
[0120]
Claims
1. A method for constructing a target tracking network structure based on optical flow compensation, characterized in that: It includes a backbone network in the form of a twin network, an optical flow-based feature compensation module and a Transformer feature fusion module, as well as a classification head and a regression head; The inputs to the backbone network in the form of the twin network are the template image and the search image, respectively. The output of the backbone network of the template image is connected to the first self-attention mechanism module of the Transformer feature fusion module, and is also connected to the optical flow-based feature compensation module. The output of the backbone network of the search image is connected to the optical flow-based feature compensation module. The Transformer feature fusion module enhances features in different ways, based on a self-attention context enhancement module and a cross-attention mechanism feature enhancement module. It includes two self-attention mechanism modules and three cross-attention mechanism modules. The first self-attention mechanism module is connected to the first and second cross-attention mechanism modules, the second self-attention module is connected to the first and second cross-attention mechanism modules, the outputs of the first and second cross-attention mechanism modules are connected to the third cross-attention mechanism module, and the output of the third cross-attention mechanism module is connected to the classification head and the regression head. The optical flow-based compensation module performs feature compensation in different ways, including local occlusion compensation and semantic-based global occlusion compensation. It includes a local compensation module and a global compensation module. The output of the local compensation module is connected to the global compensation module, and the output of the global compensation module is connected to the Transformer feature fusion module. The output of the search image backbone network is connected to the optical flow-based feature compensation module, and then to the second self-attention mechanism module of the Transformer feature fusion module.
2. The method for constructing a target tracking network structure based on optical flow compensation according to claim 1, characterized in that: The backbone network uses an improved deep neural network, ResNet50, which changes the stride of the fourth downsampling unit from 2 to 1; and changes the 3×3 convolution in the fourth layer to a dilated convolution with a stride of 2 to obtain a larger receptive field. The backbone network retains the ResNet50 network structure except for the fully connected layers, and is used to learn and extract the inherent features of the image.
3. The method for constructing a target tracking network structure based on optical flow compensation according to claim 1, characterized in that: The optical flow-based feature compensation module includes a local occlusion compensation module and a semantic-based global compensation module, which are used to compensate for image features damaged by occlusion in the search image.
4. The method for constructing a target tracking network structure based on optical flow compensation according to claim 1, characterized in that: The backbone networks of the template image and the search image are weight-shared backbone networks.
5. The method for constructing a target tracking network structure based on optical flow compensation according to claim 1, characterized in that: The Transformer feature fusion module includes a context enhancement module based on self-attention and a feature enhancement module based on cross-attention mechanism, which are used to capture similarity information between the search image and the template image.
6. The method for constructing a target tracking network structure based on optical flow compensation according to claim 1, characterized in that: The target tracking network structure is trained using a training image sample library. The training image sample library is constructed by randomly selecting the required number of images from the training image dataset, and then randomly selecting a region in each image to generate an occlusion mask. The occlusion images obtained after performing the above operation on all the selected images constitute the training image sample library.
7. The method for constructing a target tracking network structure based on optical flow compensation according to claim 6, characterized in that: The target tracking network structure is trained using samples from a training image sample library. The Adam optimization method is used to optimize the network, with optimizer parameters betas set to 0.5 and 0.999, a learning rate of 0.0005, weight_decay of 0.0001, and batch size of 20.
8. A method for implementing target tracking using a target tracking network structure based on optical flow compensation obtained by the construction method according to any one of claims 1 to 7, characterized in that... The target tracking steps are as follows: Step 1: Extract the image features of the template image and the search image through their respective backbone networks; Step 2: The template image features and search image features are fed into the optical flow-based feature compensation module. The two features are first input into the local occlusion compensation module, which calculates the forward and backward optical flow between the template image features and the search image features. The obtained forward and backward optical flows are then checked for consistency to determine the occlusion region, thereby generating an occlusion mask. The calculation formula is as follows: Where x represents the coordinates of the pixel. This indicates that a forward mapping warp operation is performed based on the calculated optical flow. This indicates that a backward mapping operation is performed on the optical flow. as well as There are two adjustment parameters; in the case of no occlusion, the equation on the left side tends to 0. If the equation is not satisfied, it can be considered that there is occlusion; according to the equation, a mask for the occluded area can be calculated. When the corresponding pixel is occluded, the value of the corresponding pixel on the mask should be 1, otherwise it should be 0. Template image features are incorporated into the occluded regions of the search image features to obtain locally compensated search image features. The calculation formula is as follows: in, To search for image features, For template image features, The search image features are those after local feature compensation; Step 3: The results of Step 2 and the template image features are processed by the global compensation module. First, the optical flow of the template image features and the locally compensated search image features is calculated. The template image features are mapped by optical flow. Then, an attention mask is used to represent the weight of important regions of the search image. The mask is used to fuse the mapped template image features and the locally compensated search image features to obtain the globally compensated search image features. The calculation formula is as follows: in, Represents the mapped template features. The search image features represent global compensation, and mask_p represents the weight mask for important regions of the search image; Step 4: The results of Step 3, namely the search image features with global feature compensation and the template image features, are fed into the first self-attention mechanism module through the Transformer feature fusion module. The calculation formula is as follows: Among them, Q, K, and V are generated by mapping the features of the template image; The output of the first self-attention mechanism is fed into both the first and second cross-attention mechanisms. The globally compensated search image features are fed into the second self-attention mechanism. The calculation method of the second self-attention mechanism is the same as that of the first self-attention mechanism, only the input is different. The output of the second self-attention mechanism is fed into both the first and second cross-attention mechanisms. The calculation formula of the first cross-attention mechanism is the same as that of the first self-attention mechanism, except that Q is generated by mapping the output of the first self-attention mechanism, and K and V are generated by mapping the output of the second self-attention mechanism. The Q of the second cross-attention mechanism is generated by mapping the output of the second self-attention mechanism, and K and V are generated by mapping the output of the first self-attention mechanism. The input of the first cross-attention mechanism and the output of the second cross-attention mechanism are fed into the third cross-attention mechanism. The output of the third cross-attention mechanism is fed into the regression head and the classification head. Step 5: The results of Step 4 are fed into the regression head and the classification head to obtain the final tracking result, i.e., the target's position coordinates.
9. The method according to claim 8, characterized in that: The specific process of obtaining tracking results using the classification head and regression head in step 5 is as follows: Step 5.1: Perform mean subtraction preprocessing on the original template image and search image, adjust the size of the template image to 128*128, and adjust the size of the search image to 256*256. When the size exceeds the original image size, fill with the average RGB value; in order to avoid adjusting the image size during training, the template image and search image of each frame are extracted offline. Step 5.2: A fine-tuned network training method is adopted. After the images are subtracted from their mean and resized, the template image and the search image are fed into the target tracking network for training. During fine-tuning, the loss function is defined as the cross-entropy function between pixels and the distance function between target locations. The classification result given by the classification head is compared with the corresponding annotations of the image at the pixel level. The classification loss function is: in, =1 indicates that this pixel is a positive sample. This represents the probability that the model predicts it to be a positive sample; the target location result given by the regression head is compared with the corresponding labeled location in the image, and the regression loss function is: This includes L1 loss and GIoU loss; =1 represents a positive sample. This represents the target bounding box predicted by the tracking model. This represents the normalized predicted target bounding box. Set to 2, Set to 5; The tracking loss function consists of classification loss and regression loss; Step 5.3: Perform several iterations of training on each pair of search images and template images, and judge the convergence of the model by the change of the loss function value until the model reaches the convergence state.
10. A computer program product, characterized in that... It includes computer-executable instructions for causing a computer to perform the method of any one of claims 8 to 9.