Visual target tracking technology based on local fine-grained target

Abstract

The invention discloses a visual target tracking technology based on a local fine-grained target. The visual target tracking technology comprises a visual target tracking system, an establishment method of the visual target tracking system and a visual target tracking method. The visual target tracking system excavates deep features between a template image containing a target to be tracked and a search image, and performs target tracking through classification and regression. The template image is input into a hierarchical self-adaptive mask mechanism module for self-adaptive local fine-grained mask processing, and a template mask image is obtained; the template image feature extraction network and the search image feature extraction network extract deep features of a template image and a search image respectively, deep fusion and relation modeling are performed on the features through the Transform encoder and the Transform decoder, and then prediction of a target bounding box in the search image is performed through classification and regression. The system can track a fine-grained target under the condition that the structure priori is not clear.

Description

Technical Field

[0001] The invention relates to a visual target tracking technology based on local fine-grained targets, and belongs to the technical field of computer vision. Background Art

[0002] Visual Object Tracking (VOT) is an important task in computer vision, which aims to continuously locate and track objects in video sequences. This technology has wide applications in many fields such as autonomous driving, human-computer interaction, and public safety. However, existing visual object tracking methods mainly focus on coarse-grained tracking of global targets, that is, tracking the entire object, while ignoring the need for local fine-grained targets. Local fine-grained object tracking aims to locate and track local parts or specific markers of an object, which is particularly important in more complex application scenarios, such as detailed obstacle monitoring in autonomous driving and gesture recognition in human-computer interaction.

[0003] Some existing studies have focused on tracking local fine-grained objects, but most of these methods focus on fine-grained objects of specific categories. References: Tom Sharp, Cem Keskin, David Robertson, John Taylor, Jamie Shotton, Toby Sharp, Ian Leichter, Andrew Vinnikov, and Antonio Criminisi. 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15). ACM, New York, NY, USA, 3633–3642. A joint system for hand tracking and pose estimation based on a depth camera is proposed. References: Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Anton Tkachenka, Georgii Sung, Clifford-Lawrence Chang, and Matthias Grundmann. 2020. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214. A hand tracking scheme using RGB modality for hand skeleton prediction in AR / VR applications was developed. Reference: Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. Ganerated hands for real-time 3D hand tracking from monocular RGB. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18). IEEE, Salt Lake City, UT, USA, 49–59. 3D hand tracking methods based on monocular RGB video have been proposed. These methods mainly rely on prior information about the target, such as known features of the human body structure, to improve the accuracy of localization.Methods based on prior information have difficulty in meeting the needs of fine-grained target tracking of arbitrary categories, especially when there is no clear structural prior. In addition, most existing methods rely on target detection technology, which achieves tracking by detecting the target position in the current frame and matching it with the position in the previous frame, but these methods usually have difficulty detecting some fine-grained targets, such as signs on objects. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a visual target tracking system based on local fine-grained targets in view of the deficiencies in the prior art, which can track fine-grained targets without clear structural priors.

[0005] Technical solution: In order to solve the above technical problems, the present invention discloses a visual target tracking system based on local fine-grained targets, including: a hierarchical adaptive mask mechanism module 11, a template image feature extraction network 12, a target encoding feature extraction module 13, a search image feature extraction network 14, a Transformer encoder 15, a Transformer decoder 16, a classification branch 17, a regression branch 18 and a search box determination module 19;

[0006] The hierarchical adaptive mask mechanism module 11 is used to perform local fine-grained mask processing on the template image to obtain a template mask image; the template image is an image containing the target to be tracked; the template image identifies the circumscribed rectangular frame of the target to be tracked;

[0007] The template image feature extraction network 12 is used to extract features from the template mask image to obtain deep template features. is the jth feature block corresponding to the i-th template image, i∈{1, 2, ..., N}, j∈{1, 2, ..., D}, N is the number of template images, and D is the number of feature blocks divided by deep template features;

[0008] The target coding feature extraction module 13 is used to extract the target coding feature according to the position information of the target to be tracked in the template image. The target coding feature corresponding to the i-th template image is tc i ;

[0009] The search image feature extraction network 14 is used to extract features from the search image to obtain deep search features. is the d-th feature block of the search image, d∈{1, 2, ..., D}; the search image is an image frame in the video sequence to be tracked;

[0010] Transformer encoder 15 is used to search for features based on the deep Deep Template Features and target encoding feature tc i Get the target embedding representation [o1, ..., o N , o test ],

[0011] [o1, ..., o N , o test ]=T en ([e1, ..., e N , e test ])

[0012] where e i Depend on With tc i Splicing to get, e test Depend on Concatenate with the search image token; o i for e i The encoding result, o test for e test The encoding result of; i∈{1, 2, ..., N};

[0013] Transformer decoder 16 is used to represent the target according to its embedding representation [o1, ..., o N , o test ] predict the weights w of the object classification and regression models in the search image;

[0014] The classification branch 17 is used to classify the target according to its embedding representation [o1, ..., o N , o test ] and the weights w of the object classification and regression models in the search image output by the Transformer decoder to predict the object classification score And score according to target classification Calculate the estimated value p of the target center point position c ;

[0015]

[0016]

[0017] Where P ij represents the coordinates of the i-th and j-th positions in the feature map, is the classification score of the position;

[0018] The regression branch 18 is used to represent the target according to its embedding [o1, ..., o N , o test ] and the weights w of the target classification and regression model in the search image output by the Transformer decoder predict the length and width of the target bounding box;

[0019] The search box determination module 19 is used to estimate the value p of the center point position of the target c The predicted bounding box of the target is calculated by the length and width of the target bounding box

[0020] The processing of the template image by the hierarchical adaptive mask mechanism module 11 comprises the following steps:

[0021] S111, setting a sliding window in the template image, wherein the sliding window slides along the longitudinal direction and the lateral direction of the template image at a preset step length respectively, to obtain a plurality of image blocks, calculating the similarity between each image block and the image within the circumscribed rectangular frame of the target to be tracked, and selecting the image block that is most similar to the image within the circumscribed rectangular frame of the target to be tracked as the coarse-grained similar image block;

[0022] S112, dividing the coarse-grained similar image block into multiple small image blocks, calculating the similarity between each small image block and the image in the circumscribed rectangular frame of the target to be tracked, and selecting the top image block that is most similar to the image in the circumscribed rectangular frame of the target to be tracked. k The pixel values ​​of the selected image block area are set to 0 to obtain the template mask image.

[0023] On the other hand, the present invention also provides a method for establishing a visual target tracking system based on local fine-grained targets, wherein the visual target tracking system is the above system, comprising the steps of:

[0024] A training set is established, wherein the sample images in the training set are all labeled with target bounding boxes; at least one sample image containing a sample and without occlusion is randomly selected from the training set without playback as a template image;

[0025] The sample images in the training set are input into the search image feature extraction network 14, and the template images are input into the hierarchical adaptive mask mechanism module 11, and the visual target tracking system is iteratively trained; the goal of the iterative training is to minimize the loss function:

[0026]

[0027] Where L c (c,p c ) is the classification loss, and the calculation formula is:

[0028]

[0029] Where S train represents the training set, |S train| is the number of samples in the training set; x represents the sample image in the training set, c is the center coordinate of the target true bounding box in x; λ is the regularization parameter used to balance the terms in the loss function to prevent overfitting; s is the target confidence score, s = x*shan, shan represents the convolutional neural network parameters of the regression branch; r(s, c) is the residual function of s and c, defined as:

[0030] r(s,c)=v c ·(m c ·s+(1-m c )·max(0,s)-y c )

[0031] Where m c is the labeled binary target mask, v c is the class-specific weighting coefficient, y c is the expected output value of the target;

[0032] is the GIoU loss, calculated as:

[0033]

[0034]

[0035] Among them, IoU means intersection over union, that is, the predicted bounding box The ratio of the intersection area and the union area of ​​the target real bounding box b, C represents the predicted bounding box and the area of ​​the minimum bounding rectangle of the target real bounding box b, Union represents the predicted bounding box and the union area of ​​the target ground-truth bounding box b;

[0036] λ c is the classification loss balance factor, λ r is the GIoU loss balancing factor.

[0037] Furthermore, the loss function also includes the foreground-background contrast loss L FBDL :

[0038]

[0039] in is the dth feature block of the search image The jth feature block corresponding to the i-th template image The cosine similarity of

[0040] w d，j is the dth feature block of the search image The jth feature block corresponding to the i-th template image The learning weights are: α is a hyperparameter that controls the smoothness of the exponential function, and rank indicates the reverse order;

[0041] The total loss function is:

[0042] λ f It is the balance factor of foreground-background contrast loss.

[0043] On the other hand, the present invention also discloses a visual target tracking method based on local fine-grained targets, comprising:

[0044] The video sequence to be tracked is input frame by frame into the search image feature extraction network 14 of the visual target tracking system, at least one image frame is selected from the video sequence to be tracked, the target to be tracked is marked in the image frame, and the selected image frame is input as a template image into the hierarchical adaptive mask mechanism module 11. The output of the search box determination module 19 is the bounding box prediction value of the target in the current frame.

[0045] On the other hand, the present invention further discloses a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the method for establishing the visual target tracking system or the method for visual target tracking is executed.

[0046] On the other hand, the present invention also discloses an electronic device, including a processor and a storage medium, wherein the storage medium is the above-mentioned computer-readable storage medium; the processor loads and executes instructions and data in the storage medium to implement the above-mentioned method for establishing the visual target tracking system, or the above-mentioned visual target tracking method.

[0047] Beneficial effects:

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. Use a hierarchical adaptive mask mechanism to process interference areas in the target template image. In the initial stage, the hierarchical adaptive mask mechanism module roughly divides the template image into blocks and selects image blocks that are most similar to the appearance features of the target area. These image blocks usually contain background areas that may be similar to the target part, which can easily interfere with the tracking results. For further accurate identification, these coarse image blocks are divided into smaller fine-grained blocks, and each small block is evaluated for similarity. The fine-grained blocks that are most similar to the target area are screened out, marked as potential interference areas, and shielded by a mask mechanism. In this adaptive way, the network can gradually eliminate interference information that may affect the tracking accuracy, thereby ensuring that the network can better focus on feature learning in the target area and reduce the negative impact of background interference on the feature extraction process.

[0050] 2. In the method for establishing a visual target tracking system based on local fine-grained targets, the present invention proposes a foreground-background distinction learning mechanism. The mechanism optimizes the learning process of the network by constructing image block pairs and calculating the feature similarity of these block pairs. Specifically, the network first generates a large number of feature block pairs between the target template and the search area, and then dynamically adjusts the learning weights according to the similarity of these block pairs. Feature block pairs with higher similarity will be assigned higher learning weights, prompting the network to more closely aggregate foreground features during training, while increasing the distance between foreground and background features. This adaptive learning method not only enhances the network's ability to distinguish between foreground and background, but also effectively deals with the problem of high-proportion background interference in the target area under unsupervised conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0052] Figure 1 A schematic diagram of a visual target tracking system based on local fine-grained targets disclosed in the present invention;

[0053] Figure 2 Schematic diagram of the similarity comparison between local fine-grained targets and the entire object;

[0054] Figure 3 Schematic diagram of the hierarchical adaptive masking mechanism. DETAILED DESCRIPTION

[0055] like Figure 1As shown, the present invention discloses a visual target tracking system 100 based on local fine-grained targets, the input of which includes one or more template images sampled from a video, and images in a video sequence. The template image includes a target to be tracked, and the circumscribed rectangular frame of the target to be tracked needs to be marked in advance; multiple template images can provide more target information, thereby improving the tracking effect. Each frame of the image in the video sequence is used as a search image, and the output of the tracking system is the predicted value of the target circumscribed frame in the search image.

[0056] The visual target tracking system includes: a hierarchical adaptive mask mechanism module 11, a template image feature extraction network 12, a target encoding feature extraction module 13, a search image feature extraction network 14, a Transformer encoder 15, a Transformer decoder 16, a classification branch 17, a regression branch 18 and a search box determination module 19.

[0057] The template image is input to the hierarchical adaptive mask mechanism module 11, and the hierarchical adaptive mask mechanism module 11 performs local fine-grained mask processing on the template image to obtain a template mask image;

[0058] Compared with the global target, the local fine-grained target has a higher similarity with the whole object, especially at the connection between the local fine-grained target and the whole object. Figure 2 In the figure, the trunk of the elephant has the same color as the whole elephant, and the similarity is high. There is no obvious dividing line between the trunk and the head of the elephant. The correlation between the target area of ​​the Box2 frame and the background area of ​​the Box1 frame is very high, and the distinction is very low. These highly interfering similar backgrounds bring additional challenges to fine-grained target tracking. When the appearance features of the matching template and the search area are similar, the highly interfering background area can easily mislead the network to think that the corresponding position in the search image is also a target, thereby reducing the tracking accuracy.

[0059] In order to alleviate this problem, the present invention uses a hierarchical adaptive mask mechanism to process the template image. The process is as follows: Figure 3 As shown. The method first uses a sliding window to evenly divide the template image containing the target, obtains several image blocks, performs a coarse-grained similarity evaluation, and then selects the background image block that is most similar to the image block containing the target for fine-grained similarity evaluation. Specifically, the hierarchical adaptive mask mechanism module 11 processes the template image including the following steps:

[0060] S111, setting a sliding window in the template image, wherein the sliding window slides along the longitudinal direction and the lateral direction of the template image at a preset step length respectively, to obtain a plurality of image blocks, calculating the similarity between each image block and the image within the circumscribed rectangular frame of the target to be tracked, and selecting the image block that is most similar to the image within the circumscribed rectangular frame of the target to be tracked as the coarse-grained similar image block;

[0061] In the present invention, the cosine distance is used as the similarity. The smaller the cosine distance is, the higher the similarity of the image blocks is.

[0062] S112, dividing the coarse-grained similar image block into multiple small image blocks, calculating the similarity between each small image block and the image in the circumscribed rectangular frame of the target to be tracked, and selecting the top image block that is most similar to the image in the circumscribed rectangular frame of the target to be tracked. k The selected image blocks are set to 0 to obtain the template mask image. k The small image block is regarded as the interference area with the highest correlation with the target in the background area, and the regional adaptive mask is realized by setting all the pixel values ​​in the area to 0 to reduce its correlation with the target area.

[0063] The hierarchical adaptive mask mechanism can reduce the influence of the connected area or the high-interference background area (object body) similar to the target area during appearance matching, so that the network can focus more on the appearance relationship modeling inside the target area. In addition, this coarse-to-fine similarity calculation method can effectively reduce the amount of similarity calculation, thereby improving tracking efficiency. Compared with the existing random mask mechanism, the hierarchical adaptive mask mechanism adopted by the present invention can adaptively select the mask area, thereby solving the influence of the connected area, and its target is more specific.

[0064] The template image feature extraction network 12 is used to extract features from the template mask image to obtain deep template features. is the jth feature block corresponding to the i-th template image, i∈{1, 2, ..., N}, j∈{1, 2, ..., D}, N is the number of template images, and D is the number of feature blocks divided by deep template features;

[0065] The target coding feature extraction module 13 is used to extract the target coding feature according to the position information of the target to be tracked in the template image. The target coding feature corresponding to the i-th template image is tc i ;

[0066] The search image feature extraction network 14 is used to extract features from the search image to obtain deep search features. is the d-th feature block of the search image, d∈{1, 2, ..., D}; the search image is an image frame in the video sequence to be tracked;

[0067] In the present invention, the template image feature extraction network 12 and the search image feature extraction network 14 have the same structure, both of which are ResNet 50 networks. The obtained deep template features and deep search features are divided into feature blocks. In the present invention, the deep template features and deep search features are divided into 3×3 feature blocks, that is, the number of feature blocks D=9. The deep template features and deep search features are sent to the Transformer codec structure, and the features are deeply fused and relationally modeled, and then the target bounding box in the search image is predicted through classification and regression.

[0068] Transformer encoder 15 is used to search for features based on the deep Deep Template Features and target encoding feature tc i Get the target embedding representation [o1, ..., o N , o test ],

[0069] [o1, ..., o N , o test ]=T en ([e1, ..., e N , e test ])

[0070] where e i Depend on With tc i Splicing obtained, Figure 1 middle represents the concatenation operation of features, e test Depend on Concatenate with the search image token; T en Represents the processing of input data by the Transformer encoder; o i for e i The encoding result, o test for e test The encoding result of; i∈{1, 2, ..., N};

[0071] Transformer decoder 16 is used to represent the target according to its embedding representation [o1, ..., o N , o test ] predict the weights w of the object classification and regression models in the search image;

[0072] The processing of data by the Transformer encoder-decoder composed of the target encoding feature extraction module 13, the Transformer encoder 15 and the Transformer decoder 16 can be found in the literature: Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. 2021. Transforming Model Prediction for Tracking. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR′21). IEEE, Nashville, TN, USA, 7095-7104.

[0073] The classification branch 17 is used to classify the target according to its embedding representation [o1, ..., o N , o test ] and the weights w of the object classification and regression models in the search image output by the Transformer decoder to predict the object classification score And score according to target classification Calculate the estimated value p of the target center point position c ;

[0074]

[0075]

[0076] where p ij represents the coordinates of the i-th and j-th positions in the feature map, is the classification score of the position;

[0077] The regression branch 18 is used to represent the target according to its embedding [o1, ..., o N , o test ] and the weights w of the target classification and regression model in the search image output by the Transformer decoder predict the length and width of the target bounding box;

[0078] In the present invention, the regression branch 18 includes: a linear layer, an attention map calculation module, and a convolutional neural network; wherein the linear layer is used to perform a linear transformation on the weight w of the target classification and regression model in the search image to obtain the weight w of the bounding box regression. reg ; The attention graph calculation module converts w reg and the target embedding representation [o1, ..., o N , otest ] in test Multiply the components to get the attention map; the attention map and o test Multiply them and input them into the convolutional neural network to get the predicted values ​​of the length and width of the target bounding box in the search image.

[0079] The search box determination module 19 is used to estimate the value p of the center point position of the target c The predicted bounding box of the target is calculated by the length and width of the target bounding box

[0080] The above visual target tracking system needs to be established according to the following training steps, including the steps:

[0081] Establishing a training set, wherein sample images in the training set are all labeled with target bounding boxes; randomly selecting at least one sample image from the training set as a template image;

[0082] The sample images in the training set are input into the search image feature extraction network 14, and the template images are input into the hierarchical adaptive mask mechanism module 11, and the visual target tracking system is iteratively trained. The loss function commonly used in iterative training is the weighted sum of the classification loss and the GIoU loss, that is, the goal of the iterative training is to minimize the loss function:

[0083] Where L c (c,p c ) is the classification loss, is the GIoU loss, λ c is the classification loss balance factor, λ r is the GIoU loss balancing factor.

[0084] Local fine-grained targets usually have narrow and irregular shapes, which makes the target occupy a smaller area in the bounding box, while the background occupies a larger proportion. Existing tracking methods usually directly extract spatial features from the bounding box as target templates, and match these template features with the features of the candidate area to locate the target. However, since the target template contains a large amount of background information, these background interferences lead to reduced feature discriminability, thereby affecting the accuracy of appearance matching. Since the foreground area usually does not have more fine-grained annotations within the target bounding box, it is difficult to use supervised constraints to guide the network to focus on learning the target area. In order to solve this problem, the present invention proposes a foreground-background contrast learning method, according to which the loss function of iterative training is improved. That is, the goal of the iterative training is to minimize the loss function:

[0085]

[0086] where λ fis the foreground-background contrast loss balance factor, L FBDL is the foreground-background contrast loss.

[0087] Based on the observation that the foreground and background are usually far apart in the feature space, it can be inferred that the greater the similarity of the feature pairs, the greater the probability that the image pairs belong to the same category. By giving similar feature pairs a larger contrast learning weight, these similar features are made closer, so as to better distinguish the foreground and background, and realize the contrast learning of foreground and background. In order to achieve this effect, the present invention sorts the image pairs in reverse order according to their similarity, and obtains the order of similarity between each search image feature block and each template image feature block. Finally, the order value is converted into weight w d，j :

[0088]

[0089] w d,j is the dth feature block of the search image The jth feature block corresponding to the i-th template image The learning weight of is between 0 and 1; α is a hyperparameter that controls the smoothness of the exponential function, and rank indicates the reverse order; is the dth feature block of the search image The jth feature block corresponding to the i-th template image The cosine similarity of

[0090] Based on the learning weights calculated above, the foreground-background contrast loss calculation formula is constructed as follows:

[0091]

[0092] This loss function adaptively shortens the distance between feature pairs, and the degree of shortening is determined by their similarity. For feature pairs with higher similarity, the loss function imposes stronger constraints to make them closer. At the same time, it does not completely discard feature pairs with lower similarity, but assigns smaller weights to them to retain the useful information that may be contained in them, which shows a high tolerance for erroneous foreground blocks. During training, the foreground-background contrast loss L FBDL The calculation of is implemented by the foreground-background contrast learning model 20. This method can not only effectively bring the foreground blocks in different images closer, but also bring the wrong foreground blocks closer. This mechanism helps to expand the difference between the foreground and the wrong foreground, thereby reducing the influence of the wrong foreground noise in the target bounding box.

[0093] Classification loss L c (c,p c) is used for the supervised classification branch, using the calculation method in the literature: Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. 2021. Transforming Model Prediction for Tracking. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR′21). IEEE, Nashville, TN, USA, 7095-7104. The calculation formula is:

[0094]

[0095] Where S train represents the training set, |S train | is the number of samples in the training set; x represents the sample image in the training set, c is the center coordinate of the target true bounding box in x; λ is the regularization parameter used to balance the terms in the loss function to prevent overfitting; s is the target confidence score, s = x*shan, shan represents the convolutional neural network parameters of the regression branch; r(s, c) is the residual function of s and c, defined as:

[0096] r(s,c)=v c ·(m c ·s+(1-m c )·max(0,s)-y c )

[0097] Where m c is the labeled binary target mask, v c is a category-specific weighting coefficient that depends on the target center coordinates and is used to adjust the loss function weights at different locations; m c and v c Continuously optimize during training. c is the expected output value of the target, that is, the true value of the target in the sample image;

[0098] GIoU loss To guide the learning of the regression branch, the generalized intersection-over-union ratio proposed in the literature: Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2019. Learning Discriminative Model Prediction for Tracking. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV'19). IEEE, Seoul, Korea, 6182-6191 is used. The calculation formula is:

[0099]

[0100]

[0101] Among them, IoU means intersection over union, that is, the predicted bounding box The ratio of the intersection area and the union area of ​​the target real bounding box b, C represents the predicted bounding box and the area of ​​the minimum bounding rectangle of the target real bounding box b, Union represents the predicted bounding box and the union area of ​​the target ground-truth bounding box b;

[0102] The total loss function L2 is c , r and λ f It serves as a balancing factor for different loss terms. Through this comprehensive loss function design, the network can be effectively optimized to achieve higher accuracy when processing complex backgrounds and fine-grained targets.

[0103] After the training is completed through the above steps, a visual target tracking system is established. Using this system, the method of tracking the target in the video includes:

[0104] The video sequence to be tracked is input frame by frame into the search image feature extraction network 14 of the visual target tracking system, at least one image frame is selected from the video sequence to be tracked, the target to be tracked is marked in the image frame, and the selected image frame is input as a template image into the hierarchical adaptive mask mechanism module 11. The output of the search box determination module 19 is the bounding box prediction value of the target in the current frame.

[0105] The bounding box of the target in each frame is obtained in sequence to form the motion trajectory of the target in the video. If the target is blocked or disappears in a frame, the target bounding box of the previous frame is used as the prediction result of the image frame where the target is lost. This ensures that the approximate position of the target can be determined even if the target is lost for a short time.

[0106] In order to verify the effect of the visual target tracking system disclosed in the present invention, this embodiment constructs a local fine-grained target tracking dataset (Local Fine-Grained Tracking dataset, LFT). The dataset consists of 64 video sequences, mainly from videos in real scenes. The video modalities include RGB modality and grayscale modality, and each frame of video in the dataset is annotated with a standard bounding box. The visual target tracking system is trained using this dataset, and the trained system is used for target tracking, and a comparative experiment is conducted with 9 classic trackers.

[0107] Nine classic trackers and the visual target tracking system proposed in this invention are evaluated on the LFT dataset. The experimental results are shown in Table 1. The AUC, Precision, sr50, and sr75 in the table represent the following: AUC (Area Under Curve), an indicator used to evaluate model performance, represents the area under the ROC curve. The larger the value, the better the model performance; Precision, represents the proportion of positive samples predicted by the model that are actually positive samples. The higher the value, the more accurate the model prediction. sr50 / sr75, the success rate when the IoU threshold is 0.5 / 0.75, indicates the proportion of the overlap between the predicted box and the true box reaching 50% / 75% or more.

[0108] Table 1 Experimental results

[0109]

[0110] As can be seen from Table 1, the visual target tracking system proposed in the present invention outperforms previous trackers in all indicators. It is particularly noteworthy that the visual target tracking system proposed in the present invention achieved 57.4% in the AUC indicator, surpassing the second-place TOMP tracker by 3.2%. This achievement is mainly due to the effectiveness of the hierarchical adaptive mask mechanism in alleviating the impact of blurred areas. In addition, the foreground-background distinction learning introduced in the present invention further enhances the network's ability to distinguish between foreground and background, thereby alleviating the challenges brought by the high background ratio in the bounding box.

[0111] According to the order from top to bottom in Table 1, the literature sources of the 9 classic trackers are as follows:

[0112] (1)Artrack:X.Wei,Y.Bai,Y.Zheng,D.Shi,and Y.Gong,“Autoregressivevisual tracking,”in Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,2023,pp.9697–9706.

[0113] (2)GRM:S.Gao,C.Zhou,and J.Zhang,“Generalized relation modeling fortransformer tracking,”in Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,2023,pp.18686–18695.

[0114] (3)Seqtrack:X.Chen,H.Peng,D.Wang,H.Lu,and H.Hu,“Seqtrack:Sequence tosequence learning for visual object tracking,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2023,pp.14572–14581.

[0115] (4)OStrack:B.Ye,H.Chang,B.Ma,S.Shan,and X.Chen,“Joint featurelearning and relation modeling for tracking:A one-stream framework,”inEuropean Conference on Computer Vision.Springer,2022,pp.341 357.

[0116] (5)AiAtrack:S.Gao,C.Zhou,C.Ma,X.Wang,and J.Yuan,“Aiatrack:Attentionin attention for transformer visual tracking,”in European Conference onComputer Vision.Springer,2022,pp.146–164.

[0117] (6)MixFormer:Y.Cui,C.Jiang,L.Wang,and G.Wu,“Mixformer:End-to-endtracking with iterative mixed attention,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2022,pp.13608–13618.

[0118] (7)TransT:B.Chen,P.Li,L.Bai,L.Qiao,Q.Shen,B.Li,W.Gan,W.Wu,andW.Ouyang,“Backbone is all your need:Asimplified architecture for visualobject tracking,”in European Conference on Computer Vision.Springer,2022,pp.375–392.

[0119] (8)SiamTrack:X.Chen,B.Yan,J.Zhu,D.Wang,X.Yang,and H.Lu,“Transformertracking,”in Proceedings of the IEEE / CVF conference on computer vision andpattern recognition,2021,pp.8126–8135.

[0120] (9)TOMP: C.Mayer, M.Danelljan, G.Bhat, M.Paul, DPPaudel, F.Yu, and L.VanGool, “Transforming model prediction for tracking,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp.8731–8740.

[0121] The present invention also discloses a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the method for establishing the visual target tracking system or the visual target tracking method is implemented.

[0122] The present invention also discloses an electronic device, including a processor and a storage medium, wherein the storage medium is a computer-readable storage medium; the processor loads and executes instructions and data in the storage medium to implement the method for establishing the above-mentioned visual target tracking system, or the above-mentioned visual target tracking method.

[0123] The present invention provides a thought and method for visual target tracking based on local fine-grained targets. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A visual target tracking system based on local fine-grained targets, characterized in that: include: Hierarchical adaptive mask mechanism module (11), template image feature extraction network (12), target encoding feature extraction module (13), search image feature extraction network (14), Transformer encoder (15), Transformer decoder (16), classification branch (17), regression branch (18) and search box determination module (19); The hierarchical adaptive mask mechanism module (11) is used to perform local fine-grained mask processing on the template image to obtain a template mask image; the template image is an image containing the target to be tracked; the template image identifies the external rectangular frame of the target to be tracked; The template image feature extraction network (12) is used to extract features from the template mask image to obtain deep template features. is the jth feature block corresponding to the i-th template image, i∈{1,2,…,N}, j∈{1,2,…,D}, N is the number of template images, and D is the number of feature blocks divided by deep template features; The target coding feature extraction module (13) is used to extract the target coding feature according to the position information of the target to be tracked in the template image. The target coding feature corresponding to the i-th template image is tc i ; The search image feature extraction network (14) is used to extract features from the search image to obtain deep search features. is the d-th feature block of the search image, d∈{1,2,…,D},; the search image is an image frame in the video sequence to be tracked; Transformer encoder (15) is used to search for features based on the deep Deep Template Features and target encoding feature tc i Get the target embedding representation [o1,…,o N ,o test ], [o1,…,o N ,o test ]=T en ([e1,…,e N ,e test ]) where e i Depend on With tc i Splicing to get, e test Depend on Concatenate with the search image token; o i for e i The encoding result, o test for e test The encoding result of; i∈{1,2,…,N}; The Transformer decoder (16) is used to represent the target according to its embedding representation [o1,…,o N ,o test ] predict the weights w of the object classification and regression models in the search image; The classification branch (17) is used to classify the target according to its embedding representation [o1,…,o N ,o test ] and the weights w of the object classification and regression models in the search image output by the Transformer decoder to predict the object classification score And score according to target classification Calculate the estimated value p of the target center point position c ; where p ij Represents the coordinates of the i,jth position in the feature map, is the classification score of the position; The regression branch (18) is used to represent the target according to its embedding [o1,…,o N ,o test ] and the weights w of the target classification and regression model in the search image output by the Transformer decoder predict the length and width of the target bounding box; The search box determination module (19) is used to estimate the value p of the target center point position. c The predicted bounding box of the target is calculated by the length and width of the target bounding box The processing of the template image by the hierarchical adaptive mask mechanism module (11) comprises the following steps: S111, setting a sliding window in the template image, wherein the sliding window slides along the longitudinal direction and the lateral direction of the template image at a preset step length respectively, to obtain a plurality of image blocks, calculating the similarity between each image block and the image within the circumscribed rectangular frame of the target to be tracked, and selecting the image block that is most similar to the image within the circumscribed rectangular frame of the target to be tracked as the coarse-grained similar image block; S112, dividing the coarse-grained similar image block into multiple small image blocks, calculating the similarity between each small image block and the image in the circumscribed rectangular frame of the target to be tracked, and selecting the top image block that is most similar to the image in the circumscribed rectangular frame of the target to be tracked. k The pixel values ​​of the selected image block area are set to 0 to obtain the template mask image.

2. The visual target tracking system according to claim 1, characterized in that: The template image feature extraction network (12) and the search image feature extraction network (14) have the same structure, both of which are ResNet 50 networks.

3. The visual target tracking system according to claim 1, characterized in that: The hierarchical adaptive mask mechanism module (11) uses cosine distance as similarity. The smaller the cosine distance, the higher the similarity.

4. The visual target tracking system according to claim 1, characterized in that: The regression branch (18) includes: a linear layer, an attention map calculation module, and a convolutional neural network; The linear layer is used to perform a linear transformation on the weights w of the target classification and regression model in the search image to obtain the weights w of the bounding box regression. reg ; The attention graph calculation module converts w reg and the target embedding representation [o1,…,o N ,o test ] in test Multiply the components to get the attention map; the attention map and o test Multiply them and input them into the convolutional neural network to get the predicted values ​​of the length and width of the target bounding box in the search image.

5. The visual target tracking system according to claim 1, characterized in that: The deep template feature is divided into 3×3 feature blocks, that is, the number of feature blocks divided by the deep template feature is D=9.

6. A method for establishing a visual target tracking system based on local fine-grained targets, wherein the visual target tracking system is a system according to any one of claims 1 to 5, characterized in that: Includes steps: A training set is established, wherein the sample images in the training set are all labeled with target bounding boxes; at least one sample image containing a sample and without occlusion is randomly selected from the training set without playback as a template image; The sample images in the training set are input into the search image feature extraction network (14), and the template images are input into the hierarchical adaptive mask mechanism module (11), and the visual target tracking system is iteratively trained; the goal of the iterative training is to minimize the loss function: Where L c (c,p c ) is the classification loss, and the calculation formula is: Where S train represents the training set, |S train | is the number of samples in the training set; x represents the sample image in the training set, c is the center coordinate of the target true bounding box in x; λ is the regularization parameter used to balance the terms in the loss function to prevent overfitting; s is the target confidence score, s = x*ω, ω represents the convolutional neural network parameters of the regression branch; r(s,c) is the residual function of s and c, defined as: r(s,c)=v c ·(m c ·s+(1-m c )·max(0,s)-y c ) Where m c is the labeled binary target mask, v c is the class-specific weighting coefficient, y c is the expected output value of the target; is the GIoU loss, calculated as: Among them, IoU means intersection over union, that is, the predicted bounding box The ratio of the intersection area and the union area of ​​the target real bounding box b, C represents the predicted bounding box and the area of ​​the minimum bounding rectangle of the target real bounding box b, Union represents the predicted bounding box and the union area of ​​the target ground-truth bounding box b; λ c is the classification loss balance factor, λ r is the GIoU loss balancing factor.

7. The method for establishing a visual target tracking system according to claim 6, characterized in that: The loss function also includes foreground-background contrast loss L FBDL : in is the dth feature block of the search image The jth feature block corresponding to the i-th template image The cosine similarity of w d,j is the dth feature block of the search image The jth feature block corresponding to the i-th template image The learning weights are: α is a hyperparameter that controls the smoothness of the exponential function, and rank indicates the reverse order; The total loss function is: λ f It is the balance factor of foreground-background contrast loss.

8. A visual target tracking method based on local fine-grained targets, characterized in that: include: Inputting the video sequence to be tracked frame by frame into the search image feature extraction network (14) of the visual target tracking system, selecting at least one image frame from the video sequence to be tracked, marking the target to be tracked in the image frame, and inputting the selected image frame as a template image into the hierarchical adaptive mask mechanism module (11), and the output of the search box determination module (19) is the bounding box prediction value of the target in the current frame; The visual target tracking system is a system established according to any one of the methods in claims 6-7.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the method for establishing a visual target tracking system according to any one of claims 6 to 7 or the visual target tracking method according to claim 8 is executed.

10. An electronic device, characterized in that: It includes a processor and a storage medium, wherein the storage medium is the computer-readable storage medium as described in claim 9; the processor loads and executes the instructions and data in the storage medium to implement the method for establishing a visual target tracking system as described in any one of claims 6-7, or the visual target tracking method as described in claim 8.

Images